Deploy on AWS EKS
Install a self-hosted deployment into your own AWS account, on an EKS cluster with a sandbox-capable node.
Audience: IT (the enabler)
This guide assumes you are comfortable operating EKS. It does not cover general Kubernetes administration.
Before you start, read Cluster and node requirements. The cluster this guide builds is not a generic one, and its storage and runtime prerequisites are the part most likely to catch you out.
Part 1: Pre-deployment bill of materials
Hand this to procurement and your cloud and security teams on day 0.
Decide these carefully, they are painful to retrofit:
- Hostname. TLS, the OIDC redirect URI, and ingress all derive from it. Changing it later means re-issuing certificates and reconfiguring your identity provider.
- Region. Moving regions later means a rebuild.
Components
| Category | Component | Spec | Notes |
|---|---|---|---|
| Compute | EKS cluster | EKS 1.36 or newer, with the cluster add-ons the platform depends on, including the EBS CSI driver | See Cluster and node requirements |
| Compute | Application node | A single x86_64 m8i.8xlarge (32 vCPU, 128 GiB) with nested virtualization explicitly enabled, in a node group of one. The instance type we primarily test against, and thus support | Hosts the application and every sandbox run. Not interchangeable with a general-purpose node group. See Node sizing |
| Storage | Node root volume | Encrypted gp3, 200 GiB | Sized for the node image and container images |
| Storage | Data root StorageClass | Created by the application chart on the EBS CSI driver: gp3, encrypted with your KMS key, WaitForFirstConsumer, expandable, reclaimPolicy: Delete | Nothing to create. Overridable with a custom class when you need tuned IOPS or throughput; see Advanced configuration |
| Storage | Data root volume | A gp3 EBS volume, 100 GiB by default, that a Pod of the application claims from that StorageClass on the node | A cache for materialized workspaces and sandbox images, not a system of record. Deleted with the node; a replacement starts cold. See The data root |
| Network | VPC and subnets | Private subnets across multiple availability zones, with NAT for egress. The application node group can span any of them | |
| Network | Cluster ingress | An ALB fronting the application, with an issued ACM certificate in the same region | You operate the ingress and terminate TLS there |
| Network | DNS record | Your chosen hostname resolving to the ingress endpoint | |
| Data | PostgreSQL | Aurora PostgreSQL Serverless v2, PostgreSQL 17 or newer, private only | Primary application data store |
| Data | S3 bucket | A bucket for the deployment, with a customer-managed KMS key | Durable storage for files, artifacts, and the workspace filesystem |
| Data | KMS key | A symmetric encryption key (the KMS default), customer-managed, with rotation enabled, in the bucket's region | Protects S3 objects, the data root volume, and stored credentials. Keep the key enabled; see Step 3 |
| Identity | OIDC IdP app | An application registered in Entra ID, Okta, or Auth0 | See Identity providers (OIDC) |
| Identity | Service account for AWS access | A Kubernetes service account in the application namespace, bound through EKS Pod Identity to an IAM role that can use the bucket and holds kms:DescribeKey, kms:GenerateDataKey, kms:Encrypt, and kms:Decrypt on the KMS key | Create it before the install and name it in the values file in Step 6 |
| Identity | EBS CSI driver key access | The IAM identity the EBS CSI driver runs as, allowed to use the KMS key | The chart-created StorageClass encrypts with your KMS key; without the grant the data root volume never provisions |
| Access | AWS and EKS admin access | Permissions for the deploying administrator, plus kubectl and helm on their workstation | EKS access entries must reference stable IAM role or user ARNs |
| Egress | Outbound HTTPS | Unrestricted egress recommended | Needed for the Sema4.ai services, the sandbox runtime's chart and image, and everything you connect the platform to. See Network endpoints |
Node sizing
The application node is a single m8i.8xlarge: 32 vCPU and 128 GiB. It is the instance type we primarily test against, and thus support. It carries the whole platform and every concurrent sandbox run, with headroom for several teams sharing the platform; see Allocatable capacity for what actually reaches the platform.
This release does not scale horizontally, so the node is the ceiling on concurrent agent work for the life of the deployment. Replacing the node costs downtime and a cold cache rather than data. See Caveats and limits.
Part 2: Deployment steps
Step 1: Pick the hostname
Choose the hostname on a domain you control, for example finance-agents.company.com. Whether it resolves publicly or only inside your network is your choice. TLS, the OIDC redirect URI, and the ingress configuration all derive from it. You point DNS at the ingress in Step 8.
Issue an ACM certificate for the hostname in the same region as the cluster, and confirm it reaches Issued before you provision.
Step 2: Register the OIDC application
Register an OIDC application in your identity provider and capture the Discovery URL, Client ID, and Client Secret.
Configure it with:
- Callback URL:
https://<hostname>/api/v1/auth/callback - Logout URL:
https://<hostname> - Application login URI:
https://<hostname>/login - Allowed web origin:
https://<hostname> - Scopes:
openid,profile,email - Client authentication: client secret in the request body (
client_secret_post)
Follow the guide for your provider: Microsoft Entra ID · Auth0 · Okta.
Step 3: Provision AWS infrastructure
Provision the infrastructure using the bill of materials as the spec. The end state must include:
- An EKS cluster with the required add-ons, including the EBS CSI driver, and
kubectlaccess configured against it. - The application node, with nested virtualization enabled and a 200 GiB root volume, in a node group of one.
- Aurora PostgreSQL, version 17 or newer, reachable from the cluster.
- An S3 bucket and a symmetric KMS key in the bucket's region.
- The application namespace, a Kubernetes service account in it, and an IAM role that can use the bucket and holds the four KMS actions below on the key, bound to the account through EKS Pod Identity. You name the account in the values file in Step 6.
- An ALB ingress and the ACM certificate from Step 1.
Node bootstrap needs nothing platform-specific: no user data, no script, no packages, and no data volume. The nodes run the stock EKS AL2023 image, and nested virtualization comes from the instance type you provision. Bootstrap does not install the sandbox runtime; that is Step 4, and it prepares nodes that join later on its own.
There is no StorageClass to create: the chart creates its own for the data root — gp3 on the EBS CSI driver, encrypted with your KMS key, meeting the contract in Storage for the data root. If you need tuned IOPS or throughput, operate your own class instead; see Advanced configuration.
The KMS key needs a grant for the EBS CSI driver. The chart-created
StorageClass encrypts the data root with your key, the managed
AmazonEBSCSIDriverPolicy grants no KMS actions, and without them the data
root claim stays Pending with a misleading InvalidVolume.NotFound error.
Grant the driver's IAM identity use of the key before you install; the exact
policy is in the
Operations reference.
Grant the application's IAM role kms:DescribeKey, kms:GenerateDataKey, kms:Encrypt, and kms:Decrypt on the KMS key.
Missing permissions can prevent startup or access to stored files.
Use the diagnostics page to check all four permissions.
Then confirm the node and the storage prerequisites:
- The node is
Ready. /dev/kvmis present on the node.kubectl get csidriver ebs.csi.aws.comreturns the driver.
Step 4: Install the sandbox runtime
Install Kata Containers into the cluster, once, following Install the sandbox runtime. Verify it before continuing:
kubectl get runtimeclass kata-clh
kubectl get nodes -l katacontainers.io/kata-runtime=trueNothing checks this at install time. The application may install without the sandbox runtime, but it will not operate correctly: agent runs fail, and the failure does not point back at the missing runtime. Make sure both commands above pass before you proceed.
Step 5: Create the database and its roles
The application does not create its own database, and it connects with three least-privilege roles rather than the Aurora master user. Connect to the cluster as the master user and run the following, with your own names and strong passwords:
CREATE ROLE <migrator_role> LOGIN PASSWORD '<migrator_password>'
NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;
CREATE ROLE <definer_role> NOLOGIN
NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION BYPASSRLS;
CREATE ROLE <app_role> LOGIN PASSWORD '<app_password>'
NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;
GRANT <definer_role> TO <migrator_role>;
CREATE DATABASE <database>;
GRANT <migrator_role> TO CURRENT_USER;
ALTER DATABASE <database> OWNER TO <migrator_role>;
GRANT CONNECT ON DATABASE <database> TO <app_role>, <definer_role>;
REVOKE <migrator_role> FROM CURRENT_USER;The migrator role applies the schema on install and upgrade, the app role is what the application connects as at runtime, and the definer role owns privileged functions and cannot log in; see The database roles for the full rationale.
Capture the database name, the three role names, and the two passwords: they go into the values file.
Step 6: Prepare your values file
Save the template as my-values.yaml and replace every REPLACE_ME before installing.
Keep the file for upgrades.
This template requires application version 3.2.5 or later.
my-values-v3-aws-eks.example.yaml# Helm values for a self-hosted deployment on AWS EKS.
# Requires application version 3.2.5 or later.
# Replace every REPLACE_ME and save as my-values.yaml for installs and upgrades.
# Application URL: https://<hostname>[:port], with no path.
# Use the same hostname in your identity provider and routing configuration.
applicationUrl: https://REPLACE_ME
# The Kubernetes service account that gives the application access to AWS
# resources. Create it in the application namespace and bind it through EKS
# Pod Identity to the IAM role for the bucket and the KMS key.
serviceAccount:
create: false
name: REPLACE_ME
# The application database and its three roles, created before the install
# (see "Create the database and its roles" in the EKS guide).
postgres:
host: REPLACE_ME
database: REPLACE_ME
appRole: REPLACE_ME
appPassword: REPLACE_ME
definerRole: REPLACE_ME
migratorRole: REPLACE_ME
migratorPassword: REPLACE_ME
# AWS resources provisioned in Step 3 of the deployment guide.
infrastructure:
platform: aws
aws:
region: REPLACE_ME
# The bucket name only. The optional key prefix keeps several
# deployments apart in one bucket.
s3BucketName: REPLACE_ME
# s3KeyPrefix: ""
# Full ARN of the symmetric KMS key for storage and stored credentials.
# Grant the application role kms:DescribeKey, kms:GenerateDataKey,
# kms:Encrypt, and kms:Decrypt. Grant the EBS CSI driver access too.
kmsKeyArn: REPLACE_ME
api:
config:
# Encrypts exported project archives. Generate with: openssl rand -hex 32
# Back up this key in your secrets manager and keep it across upgrades.
projectPortabilityKeys: '{"active":"REPLACE_ME"}'
# The OIDC application you registered against your hostname in Step 2.
auth:
oidc:
# Example discovery URLs for common providers:
# Microsoft Entra ID: https://login.microsoftonline.com/4653d2c1-77fa-412c-bf6f-db9af1f37a97/v2.0/.well-known/openid-configuration
# Okta: https://integrator-2550551.okta.com/.well-known/openid-configuration
server: REPLACE_ME # your provider's discovery URL; the bare issuer URL also works
clientId: REPLACE_ME
clientSecret: REPLACE_ME
tokenEndpointAuthMethod: client_secret_post # send the client secret in the token request body
# Ingress for the AWS Load Balancer Controller, with the ACM certificate
# auto-discovered from the host; adapt the class and annotations to your
# controller. 'scheme: internal' keeps the load balancer private; the health
# check must accept 200-404, and the idle timeout keeps websocket and
# streaming connections open for the length of an agent run.
ingress:
className: alb
host: REPLACE_ME
annotations:
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/success-codes: 200-404
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=3600The data root defaults to 100 GiB.
Increase vfs.dataRoot.size and run a Helm upgrade to expand it without a restart.
For custom storage, URLs, or ingress settings, see Advanced configuration.
Step 7: Install the application
Open the Enterprise Portal (opens in a new tab) and select the team whose license is assigned to the prod-selfhosted-v3 channel.
Follow the portal's installation flow and run its Export credentials and log in commands.
Use the my-values.yaml you prepared above and skip the preflight plugin step.
Install the latest available version, replacing <release-name> with the name you want to give this deployment and <your-namespace> with your application namespace:
helm install <release-name> \
oci://registry.sema4.ai/sema4ai/prod-selfhosted-v3/blockparty \
--values my-values.yaml \
--namespace <your-namespace>Omit --version to install the latest version available on the v3 channel.
If you need a specific release, add --version <chart-version> using the exact chart version shown in the portal.
Keep the release name, namespace, and values file for future upgrades.
Expect the VFS and sandbox Pods to sit in ContainerCreating for a few minutes on a first install: they bind the data root, and they start once the data-root workload has claimed, formatted, and mounted it.
Step 8: Point DNS at your ingress
Create the DNS record resolving your hostname to the ALB endpoint. Sign-in does not complete until the hostname resolves and TLS is serving.
Step 9: Validate
- Check the data root: the claim is
Boundand thedata-rootlog ends withdata root ready. See Verify the data root. - Browse to
https://<hostname>. Sign-in should redirect to your identity provider. - Sign in with a permitted user and confirm you land in the workspace.
- Choose the workspace's models. Agents, workflows, and Sai do not run until the workspace has a Primary and a Fast model, and every page shows the banner Choose this workspace's models until then. Click Choose models, or go to Configuration > Models, choose both models, and click Save models. The Image model is optional.
The first user of a new deployment becomes its admin. Sign in yourself first, before opening access more widely.
Step 10: Protect your keys and backups
- Back up the database.
- Keep the KMS key enabled and accessible for as long as you retain the database or its backups.
- Store
api.config.projectPortabilityKeysin your secrets manager so you can decrypt exported project archives.
Permanently deleting the KMS key makes stored credentials unrecoverable, even from a database backup. Disabling it prevents the application from accessing those credentials until access is restored.
Next steps
To upgrade this deployment, follow Update the application.
With the application running, continue to Administration to manage model platforms, connect integrations and data sources, and set up teams. For day-2 behavior (resizing the data root, replacing the node, upgrading the sandbox runtime), see the Operations reference.