Operations reference

How the deployment behaves in operation, and how to diagnose the failures that do not explain themselves. Nothing on this page is needed for a first install; the deployment guides link here where the detail applies.

Audience: IT (the enabler)

The application node

Allocatable capacity

Not all of the instance's capacity reaches the platform. Cluster add-ons and system DaemonSets take their share first, and the platform's own baseline requests come out of what is left before the first sandbox run starts. The sandbox runner pool is sized against that allocatable capacity, not the instance label.

Replacing the node

Replacing the application node costs downtime and a cold cache, not data. The replacement can join wherever the node group spans, the sandbox runtime prepares it automatically as it joins, and a fresh data root is provisioned there and refills from your bucket; the old node's data root volume is deleted with its Pod. Expect slower runs while the caches refill.

In-place reboots

An in-place node reboot briefly deregisters the sandbox runtime. Amazon Linux 2023 nodes regenerate /etc/containerd/config.toml at every boot, which drops the registration the kata-deploy chart added; the chart's DaemonSet restores it when its Pod restarts after the reboot, restarting containerd in the process. Expect a short window after a reboot in which sandbox Pods wait in ContainerCreating with FailedCreatePodSandBox events naming the kata-clh handler. It closes on its own, with no node configuration and no operator action.

The data root

The mount contract

You do not have to meet this contract yourself; the platform's data-root workload does. It is listed here so you can recognize a healthy mount, and because it is what the sandbox runtime and the kubelet depend on.

The mount options are noatime, ssd, space_cache=v2, discard=async, compress=zstd:3, and user_subvol_rm_allowed.

Resizing the data root

vfs.dataRoot.size is live. Raise it in the values file and upgrade: within a couple of minutes each node's data-root Pod patches its own claim, the CSI driver expands the volume in place, and the platform grows the filesystem within about a minute of the device reporting its new size. Nothing restarts, no workload notices, and a node that joins later gets the new size directly. On EBS, consecutive grows wait out AWS's limit of one modification per volume per six hours.

Shrinking is not possible. Kubernetes cannot reduce a claim, so the platform never sends a smaller request. The upgrade itself still succeeds; the data-root Pod logs an error every minute until the value is raised back to at least the current size, and that log line is the only signal, so treat a smaller value as a values-file mistake to revert rather than something the platform will apply.

Changing the StorageClass

On AWS the chart creates and names the data root's class itself from the infrastructure block; setting vfs.dataRoot.storageClassName replaces it with a class you operate. Changing vfs.dataRoot.storageClassName never touches an existing volume. A claim's class is fixed at creation, so a new class applies only to data roots created afterwards: on nodes that join later, and on the replacement when a node is replaced. Until then the deployment runs with a mix of old-class and new-class volumes, the upgrade succeeds without warning about it, and the STORAGECLASS column of the claims is how you see which class each data root actually uses. To move a node deliberately, replace the node; the cost is that node's cache.

StorageClass parameters behave the same way: they are immutable, so a parameter change means a new class, and it affects only volumes created afterwards. Materialization is reflink- and metadata-heavy, so if runs are slow, raise IOPS before throughput.

The built-in gp2 class

A standard EKS cluster ships a default gp2 StorageClass that the EBS CSI driver serves, and it provisions Block volumes with WaitForFirstConsumer binding, so it works for the data root with trade-offs: it does not allow expansion, so the data root size is fixed for the life of the volume; it is gp2 burst IOPS rather than provisioned gp3; and the volume is encrypted only if your account enables EBS encryption by default. If those limits suit you, set storageClassName: gp2 in the values file and skip creating a class. A cluster without a default class, or whose default is not EBS, needs its own class either way.

A claim that stays Pending: the KMS grant

When the StorageClass encrypts with a customer-managed key, the EBS CSI driver needs KMS permissions. The managed AmazonEBSCSIDriverPolicy covers the EC2 volume API only and grants no KMS actions. Without them the claim stays Pending and the driver reports InvalidVolume.NotFound, a misleading symptom whose real cause, kms:GenerateDataKeyWithoutPlaintext denied, appears only in CloudTrail.

Attach a policy to the IAM identity the driver runs as (the node role when the add-on uses node credentials, otherwise the driver's own Pod Identity role) that allows kms:Encrypt, kms:Decrypt, kms:ReEncrypt*, kms:GenerateDataKey*, and kms:DescribeKey on the key, plus kms:CreateGrant, kms:ListGrants, and kms:RevokeGrant with the condition kms:GrantIsForAWSResource: true.

A claim that hung before the fix does not recover on its own: delete it, and the next helm upgrade recreates it.

The database roles

The application connects with three least-privilege roles rather than the master user:

• The migrator role owns the database and applies the schema on install and on every upgrade. It enables the pgcrypto and citext extensions itself, which owning the database allows.
• The definer role cannot log in. It owns the functions that run with elevated privileges, and it is the only role that bypasses row-level security.
• The app role is what the application connects as at runtime.

Roles are server-wide in PostgreSQL, so give each deployment its own set if several share one database cluster.

The sandbox runtime

Why you install it, and not the platform

Kata is a node-level runtime. It puts binaries on every node, reconfigures containerd, and registers a cluster-scoped RuntimeClass. That makes it cluster infrastructure, on the same footing as your CNI or your CSI driver, and it belongs to whoever owns the cluster:

• It is installed once per cluster, however many deployments share it. The RuntimeClass is a cluster-scoped object, so it cannot belong to one release. Do not tie the installation to one deployment's lifecycle: uninstalling that deployment would take the runtime away from the others.
• Its upgrades are node maintenance. Installing or upgrading it restarts the container runtime on every node it prepares. That is a window you schedule, independently of platform upgrades, paced with the chart's updateStrategy.
• Nodes that join later are handled for you. The chart runs as a DaemonSet, so a node appearing after the install, such as a replacement for the application node, is prepared when it joins.

What the pinned values do

Two settings in the install values are not optional, because the platform depends on them:

• The Go Cloud Hypervisor shim (clh), not clh-runtime-rs. The sandbox reaches each run's control channel through a socket the Go runtime places at a specific path. The Rust runtime lays that state out differently, and runs will not start.
• --xattr alongside the upstream default --announce-submounts in virtio_fs_extra_args. Each run's workspace is composed from nested mounts that the guest has to observe, and the guest executes a helper that carries Linux file capabilities, which requires the filesystem daemon to expose security.capability.

The rest narrows a general-purpose chart to what a deployment needs: one shim instead of twenty, no alternative snapshotter, no monitoring sidecar, and no bundled node-feature-discovery, which would collide with an existing installation.

overheadEnabled: false is worth understanding. The chart can advertise a fixed per-pod CPU and memory overhead on the RuntimeClass. The platform already accounts for sandbox cost in its own admission budgets, so leaving it on charges the same capacity twice and reduces how many runs your node admits.

What this article will cover

• Recovery procedure for a lost node
• What the cache holds, and how a cold node behaves while it refills
• What a full data root looks like, and the alerting to pair with it
• Troubleshooting a data root that never binds or never mounts
• Troubleshooting a node the sandbox runtime chart could not prepare