Deploy on AWS EKS
Install a self-hosted deployment into your own AWS account, on an EKS cluster with a sandbox-capable node.
Audience: IT (the enabler)
This guide assumes you are comfortable operating EKS. It does not cover general Kubernetes administration.
Before you start, read Cluster and node requirements. The cluster this guide builds is not a generic one, and its storage and runtime prerequisites are the part most likely to catch you out.
Part 1: Pre-deployment bill of materials
Hand this to procurement and your cloud and security teams on day 0.
Decide these carefully, they are painful to retrofit:
- Hostname. TLS, the OIDC redirect URI, and ingress all derive from it. Changing it later means re-issuing certificates and reconfiguring your identity provider.
- Region. Moving regions later means a rebuild.
Components
| Category | Component | Spec | Notes |
|---|---|---|---|
| Compute | EKS cluster | EKS 1.36 or newer, with the cluster add-ons the platform depends on, including the EBS CSI driver | See Cluster and node requirements |
| Compute | Application node | A single x86_64 m8i.8xlarge (32 vCPU, 128 GiB) with nested virtualization explicitly enabled, in a node group of one. The instance type we primarily test against, and thus support | Hosts the application and every sandbox run. Not interchangeable with a general-purpose node group. See Node sizing |
| Storage | Node root volume | Encrypted gp3 | Sized for the node image and container images |
| Storage | Data root StorageClass | Created by the application chart on the EBS CSI driver: gp3, encrypted with your KMS key, WaitForFirstConsumer, expandable, reclaimPolicy: Delete | Nothing to create. Overridable with a custom class when you need tuned IOPS or throughput; see Advanced configuration |
| Storage | Data root volume | A gp3 EBS volume, 100 GiB by default, that a Pod of the application claims from that StorageClass on the node | A cache for materialized workspaces and sandbox images, not a system of record. Deleted with the node; a replacement starts cold. See The data root |
| Network | VPC and subnets | Private subnets across multiple availability zones, with NAT for egress. The application node group can span any of them | |
| Network | Cluster ingress | An ALB fronting the application, with an issued ACM certificate in the same region | You operate the ingress and terminate TLS there |
| Network | DNS record | Your chosen hostname resolving to the ingress endpoint | |
| Data | PostgreSQL | Aurora PostgreSQL Serverless v2, PostgreSQL 17 or newer, private only | Primary application data store |
| Data | S3 bucket | A bucket for the deployment, with a customer-managed KMS key | Durable storage for files, artifacts, and the workspace filesystem |
| Data | KMS key | Customer-managed, with rotation enabled | Envelope encryption of secrets at rest, and encryption of the data root volume |
| Identity | OIDC IdP app | An application registered in Entra ID, Okta, or Auth0 | See Identity providers (OIDC) |
| Identity | Service account for AWS access | A Kubernetes service account in the application namespace, bound through EKS Pod Identity to an IAM role that can use the bucket and the KMS key | Create it before the install and name it in the values file in Step 6 |
| Identity | EBS CSI driver key access | The IAM identity the EBS CSI driver runs as, allowed to use the KMS key | The chart-created StorageClass encrypts with your KMS key; without the grant the data root volume never provisions |
| Access | AWS and EKS admin access | Permissions for the deploying administrator, plus kubectl and helm on their workstation | EKS access entries must reference stable IAM role or user ARNs |
| Egress | Outbound HTTPS | Unrestricted egress recommended | Needed for the Sema4.ai services, the sandbox runtime's chart and image, and everything you connect the platform to. See Network endpoints |
Node sizing
The application node is a single m8i.8xlarge: 32 vCPU and 128 GiB. It is the instance type we primarily test against, and thus support. It carries the whole platform and every concurrent sandbox run, with headroom for several teams sharing the platform; see Allocatable capacity for what actually reaches the platform.
This release does not scale horizontally, so the node is the ceiling on concurrent agent work for the life of the deployment. Replacing the node costs downtime and a cold cache rather than data. See Caveats and limits.
Part 2: Deployment steps
Step 1: Pick the hostname
Choose the hostname on a domain you control, for example finance-agents.company.com. Whether it resolves publicly or only inside your network is your choice. TLS, the OIDC redirect URI, and the ingress configuration all derive from it. You point DNS at the ingress in Step 8.
Issue an ACM certificate for the hostname in the same region as the cluster, and confirm it reaches Issued before you provision.
Step 2: Register the OIDC application
Register an OIDC application in your identity provider and capture the Discovery URL, Client ID, and Client Secret.
Configure it with:
- Callback URL:
https://<hostname>/api/v1/auth/callback - Logout URL:
https://<hostname> - Application login URI:
https://<hostname>/login - Allowed web origin:
https://<hostname> - Scopes:
openid,profile,email
Follow the guide for your provider: Microsoft Entra ID · Auth0 · Okta.
Step 3: Provision AWS infrastructure
Provision the infrastructure using the bill of materials as the spec. The end state must include:
- An EKS cluster with the required add-ons, including the EBS CSI driver, and
kubectlaccess configured against it. - The application node, with nested virtualization enabled, in a node group of one.
- Aurora PostgreSQL, version 17 or newer, reachable from the cluster.
- An S3 bucket and a KMS key.
- The application namespace, a Kubernetes service account in it, and an IAM role that can use the bucket and the key, bound to the account through EKS Pod Identity. You name the account in the values file in Step 6.
- An ALB ingress and the ACM certificate from Step 1.
Node bootstrap needs nothing platform-specific: no user data, no script, no packages, and no data volume. The nodes run the stock EKS AL2023 image, and nested virtualization comes from the instance type you provision. Bootstrap does not install the sandbox runtime; that is Step 4, and it prepares nodes that join later on its own.
There is no StorageClass to create: the chart creates its own for the data root — gp3 on the EBS CSI driver, encrypted with your KMS key, meeting the contract in Storage for the data root. If you need tuned IOPS or throughput, operate your own class instead; see Advanced configuration.
The KMS key needs a grant for the EBS CSI driver. The chart-created
StorageClass encrypts the data root with your key, the managed
AmazonEBSCSIDriverPolicy grants no KMS actions, and without them the data
root claim stays Pending with a misleading InvalidVolume.NotFound error.
Grant the driver's IAM identity use of the key before you install; the exact
policy is in the
Operations reference.
Then confirm the node and the storage prerequisites:
- The node is
Ready. /dev/kvmis present on the node.kubectl get csidriver ebs.csi.aws.comreturns the driver.
Step 4: Install the sandbox runtime
Install Kata Containers into the cluster, once, following Install the sandbox runtime. Verify it before continuing:
kubectl get runtimeclass kata-clh
kubectl get nodes -l katacontainers.io/kata-runtime=trueNothing checks this at install time. The application may install without the sandbox runtime, but it will not operate correctly: agent runs fail, and the failure does not point back at the missing runtime. Make sure both commands above pass before you proceed.
Step 5: Create the database and its roles
The application does not create its own database, and it connects with three least-privilege roles rather than the Aurora master user. Connect to the cluster as the master user and run the following, with your own names and strong passwords:
CREATE ROLE <migrator_role> LOGIN PASSWORD '<migrator_password>'
NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;
CREATE ROLE <definer_role> NOLOGIN
NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION BYPASSRLS;
CREATE ROLE <app_role> LOGIN PASSWORD '<app_password>'
NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;
GRANT <definer_role> TO <migrator_role>;
CREATE DATABASE <database>;
GRANT <migrator_role> TO CURRENT_USER;
ALTER DATABASE <database> OWNER TO <migrator_role>;
REVOKE <migrator_role> FROM CURRENT_USER;
GRANT CONNECT ON DATABASE <database> TO <app_role>, <definer_role>;The migrator role applies the schema on install and upgrade, the app role is what the application connects as at runtime, and the definer role owns privileged functions and cannot log in; see The database roles for the full rationale.
Capture the database name, the three role names, and the two passwords: they go into the values file.
Step 6: Prepare your values file
The values file is your readiness check. Complete it before you install.
Every value in it is something a prerequisite from Part 1 produced, so a value
you cannot fill in is a prerequisite that is not ready. Do not start Step 7
with a REPLACE_ME left in the file.
Start from the template below and replace every REPLACE_ME. Only what the chart cannot know about your environment is in it; everything else is a chart default, tuned for the supported node. Keep the file: you reuse it for every upgrade.
my-values-v3-aws-eks.example.yaml# Helm values for a self-hosted deployment on AWS EKS.
#
# Complete every REPLACE_ME before you install; a value you cannot fill in is
# a prerequisite that is not ready yet. Keep this file: you reuse it for
# upgrades. Everything not set here is a chart default, tuned for the
# supported 32 vCPU / 128 GiB node.
# The Kubernetes service account that gives the application access to AWS
# resources. Create it in the application namespace and bind it through EKS
# Pod Identity to the IAM role for the bucket and the KMS key.
serviceAccount:
create: false
name: REPLACE_ME
# The application database and its three roles, created before the install
# (see "Create the database and its roles" in the EKS guide).
postgres:
host: REPLACE_ME
database: REPLACE_ME
appRole: REPLACE_ME
appPassword: REPLACE_ME
definerRole: REPLACE_ME
migratorRole: REPLACE_ME
migratorPassword: REPLACE_ME
# The AWS resources from the bill of materials, in one block. From these the
# chart derives everything AWS-shaped:
# - object storage at s3://<bucket>[/<prefix>], reached through the service
# account's IAM role (no static keys);
# - a gp3 StorageClass on the EBS CSI driver, encrypted with your KMS key,
# and the data root claimed from it as a raw block volume on the node
# (100 GiB by default; raise vfs.dataRoot.size later and upgrade, the
# filesystem grows without a restart);
# - the AWS region for SDK calls (the IAM role supplies credentials, not a
# region);
# - the sandbox wiring for EKS: the Kata runtime you installed before the
# platform, reached through the node's containerd, with the release
# installing no Kata of its own.
# Anything set explicitly elsewhere in this file wins over a derived value.
infrastructure:
platform: aws
aws:
region: REPLACE_ME
# The bucket name only. The optional key prefix keeps several
# deployments apart in one bucket.
s3BucketName: REPLACE_ME
# s3KeyPrefix: ""
# Full ARN of the KMS key that encrypts the data root volumes. The EBS
# CSI driver's IAM identity must be granted use of this key (see the KMS
# grant callout in the EKS guide) or the volume never provisions.
kmsKeyArn: REPLACE_ME
api:
config:
# Encryption keys, yours to generate and to keep. Generate a long random
# string for each, for example with `openssl rand -hex 32`.
# secretsKeys encrypts the credentials the platform stores in its
# database: model platform keys, integration and OAuth tokens. The
# database outlives the cluster, and without the exact key its encrypted
# contents cannot be decrypted — the database may become unusable.
# projectPortabilityKeys protects exported project archives the same
# way. Keep this file backed up somewhere safe outside the cluster.
secretsKeys: '{"v1":"REPLACE_ME"}'
projectPortabilityKeys: '{"active":"REPLACE_ME"}'
# Your hostname as https://<hostname>, and the OIDC application you
# registered against it.
auth:
allowedOrigins: '["https://REPLACE_ME"]'
webAppUrl: https://REPLACE_ME
oidc:
server: REPLACE_ME # your provider's discovery URL; the bare issuer URL also works
clientId: REPLACE_ME
clientSecret: REPLACE_ME
redirectUri: https://REPLACE_ME/api/v1/auth/callback
realtime:
webhookPublicBaseUrl: https://REPLACE_ME
# Ingress for the AWS Load Balancer Controller, with the ACM certificate
# auto-discovered from the host; adapt the class and annotations to your
# controller. 'scheme: internal' keeps the load balancer private; the health
# check must accept 200-404, and the idle timeout keeps websocket and
# streaming connections open for the length of an agent run.
ingress:
className: alb
host: REPLACE_ME
annotations:
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/success-codes: 200-404
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=3600Three things are worth a moment. The data root size is not in the file because 100 GiB is the chart default; you can raise it later (vfs.dataRoot.size) without a restart, so start small. Every value the infrastructure block derives has an explicit override that wins over the convention — a StorageClass of your own, or vfs.blobs.uri set directly — but the file above is the supported shape; see Advanced configuration. The data root mounts at /var/lib/blockparty/<release>, named after the Helm release you install under; nothing in the file needs to say so. And no internal service tokens are in the file: the chart generates them at first install, reuses them across upgrades, and they die with the deployment. The two encryption keys are in the file, deliberately — they protect what the platform stores in your database, so their custody stays with you and this file, not with the cluster.
Step 7: Install the application
The self-hosted release channel for version 3 is not open yet. There is no install command to publish. This step will carry the registry login and the chart install once the channel ships.
Expect the VFS and sandbox Pods to sit in ContainerCreating for a few minutes on a first install: they bind the data root, and they start once the data-root workload has claimed, formatted, and mounted it.
Step 8: Point DNS at your ingress
Create the DNS record resolving your hostname to the ALB endpoint. Sign-in does not complete until the hostname resolves and TLS is serving.
Step 9: Validate
- Check the data root: the claim is
Boundand thedata-rootlog ends withdata root ready. See Verify the data root. - Browse to
https://<hostname>. Sign-in should redirect to your identity provider. - Sign in with a permitted user and confirm you land in the workspace.
- Run a smoke test that exercises a sandboxed run, which confirms the data root, the sandbox runtime, the database, and object storage together. Expect the first run to be slower than steady state while the caches fill.
The first user of a new deployment becomes its owner. Sign in yourself first, before opening access more widely.
Step 10: Back up the encryption keys
The two encryption keys you generated into the values file encrypt the secrets the platform stores in its database — model platform credentials, integration and OAuth tokens — and the database outlives the cluster.
Anything encrypted with a lost key cannot be read back. Stored credentials and connections cannot be recovered, and the database may become unusable. Treat the keys as irreplaceable.
Store the two key values in your secrets manager, alongside your other infrastructure credentials, so they survive the loss of the machine that holds the values file. Treat them as root credentials.
Next steps
With the application running, continue to Administration to configure models, connect integrations and data sources, and set up teams. For day-2 behavior — resizing the data root, replacing the node, upgrading the sandbox runtime — see the Operations reference.
What this article will cover
- The full bill of materials with concrete sizes, versions, and SKUs
- The install commands, once the self-hosted channel ships
- Screenshots of the install flow
- Upgrading an existing deployment
- Teardown, and what survives it