Version 3
AWS EKS

Deploy on AWS EKS

Install a self-hosted deployment into your own AWS account, on an EKS cluster with a sandbox-capable node.

Audience: IT (the enabler)

This guide assumes you are comfortable operating EKS. It does not cover general Kubernetes administration.

Before you start, read Cluster and node requirements. The cluster this guide builds is not a generic one, and its storage and runtime prerequisites are the part most likely to catch you out.

Part 1: Pre-deployment bill of materials

Hand this to procurement and your cloud and security teams on day 0.

Decide these carefully, they are painful to retrofit:

  • Hostname. TLS, the OIDC redirect URI, and ingress all derive from it. Changing it later means re-issuing certificates and reconfiguring your identity provider.
  • Region. Moving regions later means a rebuild.

Components

CategoryComponentSpecNotes
ComputeEKS clusterEKS 1.36 or newer, with the cluster add-ons the platform depends on, including the EBS CSI driverSee Cluster and node requirements
ComputeApplication nodeA single x86_64 m8i.8xlarge (32 vCPU, 128 GiB) with nested virtualization explicitly enabled, in a node group of one. The instance type we primarily test against, and thus supportHosts the application and every sandbox run. Not interchangeable with a general-purpose node group. See Node sizing
StorageNode root volumeEncrypted gp3, 200 GiBSized for the node image and container images
StorageData root StorageClassCreated by the application chart on the EBS CSI driver: gp3, encrypted with your KMS key, WaitForFirstConsumer, expandable, reclaimPolicy: DeleteNothing to create. Overridable with a custom class when you need tuned IOPS or throughput; see Advanced configuration
StorageData root volumeA gp3 EBS volume, 100 GiB by default, that a Pod of the application claims from that StorageClass on the nodeA cache for materialized workspaces and sandbox images, not a system of record. Deleted with the node; a replacement starts cold. See The data root
NetworkVPC and subnetsPrivate subnets across multiple availability zones, with NAT for egress. The application node group can span any of them
NetworkCluster ingressAn ALB fronting the application, with an issued ACM certificate in the same regionYou operate the ingress and terminate TLS there
NetworkDNS recordYour chosen hostname resolving to the ingress endpoint
DataPostgreSQLAurora PostgreSQL Serverless v2, PostgreSQL 17 or newer, private onlyPrimary application data store
DataS3 bucketA bucket for the deployment, with a customer-managed KMS keyDurable storage for files, artifacts, and the workspace filesystem
DataKMS keyA symmetric encryption key (the KMS default), customer-managed, with rotation enabled, in the bucket's regionProtects S3 objects, the data root volume, and stored credentials. Keep the key enabled; see Step 3
IdentityOIDC IdP appAn application registered in Entra ID, Okta, or Auth0See Identity providers (OIDC)
IdentityService account for AWS accessA Kubernetes service account in the application namespace, bound through EKS Pod Identity to an IAM role that can use the bucket and holds kms:DescribeKey, kms:GenerateDataKey, kms:Encrypt, and kms:Decrypt on the KMS keyCreate it before the install and name it in the values file in Step 6
IdentityEBS CSI driver key accessThe IAM identity the EBS CSI driver runs as, allowed to use the KMS keyThe chart-created StorageClass encrypts with your KMS key; without the grant the data root volume never provisions
AccessAWS and EKS admin accessPermissions for the deploying administrator, plus kubectl and helm on their workstationEKS access entries must reference stable IAM role or user ARNs
EgressOutbound HTTPSUnrestricted egress recommendedNeeded for the Sema4.ai services, the sandbox runtime's chart and image, and everything you connect the platform to. See Network endpoints

Node sizing

The application node is a single m8i.8xlarge: 32 vCPU and 128 GiB. It is the instance type we primarily test against, and thus support. It carries the whole platform and every concurrent sandbox run, with headroom for several teams sharing the platform; see Allocatable capacity for what actually reaches the platform.

This release does not scale horizontally, so the node is the ceiling on concurrent agent work for the life of the deployment. Replacing the node costs downtime and a cold cache rather than data. See Caveats and limits.

Part 2: Deployment steps

Step 1: Pick the hostname

Choose the hostname on a domain you control, for example finance-agents.company.com. Whether it resolves publicly or only inside your network is your choice. TLS, the OIDC redirect URI, and the ingress configuration all derive from it. You point DNS at the ingress in Step 8.

Issue an ACM certificate for the hostname in the same region as the cluster, and confirm it reaches Issued before you provision.

Step 2: Register the OIDC application

Register an OIDC application in your identity provider and capture the Discovery URL, Client ID, and Client Secret.

Configure it with:

  • Callback URL: https://<hostname>/api/v1/auth/callback
  • Logout URL: https://<hostname>
  • Application login URI: https://<hostname>/login
  • Allowed web origin: https://<hostname>
  • Scopes: openid, profile, email
  • Client authentication: client secret in the request body (client_secret_post)

Follow the guide for your provider: Microsoft Entra ID · Auth0 · Okta.

Step 3: Provision AWS infrastructure

Provision the infrastructure using the bill of materials as the spec. The end state must include:

  • An EKS cluster with the required add-ons, including the EBS CSI driver, and kubectl access configured against it.
  • The application node, with nested virtualization enabled and a 200 GiB root volume, in a node group of one.
  • Aurora PostgreSQL, version 17 or newer, reachable from the cluster.
  • An S3 bucket and a symmetric KMS key in the bucket's region.
  • The application namespace, a Kubernetes service account in it, and an IAM role that can use the bucket and holds the four KMS actions below on the key, bound to the account through EKS Pod Identity. You name the account in the values file in Step 6.
  • An ALB ingress and the ACM certificate from Step 1.

Node bootstrap needs nothing platform-specific: no user data, no script, no packages, and no data volume. The nodes run the stock EKS AL2023 image, and nested virtualization comes from the instance type you provision. Bootstrap does not install the sandbox runtime; that is Step 4, and it prepares nodes that join later on its own.

There is no StorageClass to create: the chart creates its own for the data root — gp3 on the EBS CSI driver, encrypted with your KMS key, meeting the contract in Storage for the data root. If you need tuned IOPS or throughput, operate your own class instead; see Advanced configuration.

The KMS key needs a grant for the EBS CSI driver. The chart-created StorageClass encrypts the data root with your key, the managed AmazonEBSCSIDriverPolicy grants no KMS actions, and without them the data root claim stays Pending with a misleading InvalidVolume.NotFound error. Grant the driver's IAM identity use of the key before you install; the exact policy is in the Operations reference.

Grant the application's IAM role kms:DescribeKey, kms:GenerateDataKey, kms:Encrypt, and kms:Decrypt on the KMS key. Missing permissions can prevent startup or access to stored files. Use the diagnostics page to check all four permissions.

Then confirm the node and the storage prerequisites:

  • The node is Ready.
  • /dev/kvm is present on the node.
  • kubectl get csidriver ebs.csi.aws.com returns the driver.

Step 4: Install the sandbox runtime

Install Kata Containers into the cluster, once, following Install the sandbox runtime. Verify it before continuing:

kubectl get runtimeclass kata-clh
kubectl get nodes -l katacontainers.io/kata-runtime=true

Nothing checks this at install time. The application may install without the sandbox runtime, but it will not operate correctly: agent runs fail, and the failure does not point back at the missing runtime. Make sure both commands above pass before you proceed.

Step 5: Create the database and its roles

The application does not create its own database, and it connects with three least-privilege roles rather than the Aurora master user. Connect to the cluster as the master user and run the following, with your own names and strong passwords:

CREATE ROLE <migrator_role> LOGIN PASSWORD '<migrator_password>'
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;
CREATE ROLE <definer_role> NOLOGIN
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION BYPASSRLS;
CREATE ROLE <app_role> LOGIN PASSWORD '<app_password>'
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;
GRANT <definer_role> TO <migrator_role>;
CREATE DATABASE <database>;
GRANT <migrator_role> TO CURRENT_USER;
ALTER DATABASE <database> OWNER TO <migrator_role>;
GRANT CONNECT ON DATABASE <database> TO <app_role>, <definer_role>;
REVOKE <migrator_role> FROM CURRENT_USER;

The migrator role applies the schema on install and upgrade, the app role is what the application connects as at runtime, and the definer role owns privileged functions and cannot log in; see The database roles for the full rationale.

Capture the database name, the three role names, and the two passwords: they go into the values file.

Step 6: Prepare your values file

Save the template as my-values.yaml and replace every REPLACE_ME before installing. Keep the file for upgrades. This template requires application version 3.2.5 or later.

Download my-values-v3-aws-eks.example.yaml
# Helm values for a self-hosted deployment on AWS EKS.
# Requires application version 3.2.5 or later.
# Replace every REPLACE_ME and save as my-values.yaml for installs and upgrades.
 
# Application URL: https://<hostname>[:port], with no path.
# Use the same hostname in your identity provider and routing configuration.
applicationUrl: https://REPLACE_ME
 
# The Kubernetes service account that gives the application access to AWS
# resources. Create it in the application namespace and bind it through EKS
# Pod Identity to the IAM role for the bucket and the KMS key.
serviceAccount:
  create: false
  name: REPLACE_ME
 
# The application database and its three roles, created before the install
# (see "Create the database and its roles" in the EKS guide).
postgres:
  host: REPLACE_ME
  database: REPLACE_ME
  appRole: REPLACE_ME
  appPassword: REPLACE_ME
  definerRole: REPLACE_ME
  migratorRole: REPLACE_ME
  migratorPassword: REPLACE_ME
 
# AWS resources provisioned in Step 3 of the deployment guide.
infrastructure:
  platform: aws
  aws:
    region: REPLACE_ME
    # The bucket name only. The optional key prefix keeps several
    # deployments apart in one bucket.
    s3BucketName: REPLACE_ME
    # s3KeyPrefix: ""
    # Full ARN of the symmetric KMS key for storage and stored credentials.
    # Grant the application role kms:DescribeKey, kms:GenerateDataKey,
    # kms:Encrypt, and kms:Decrypt. Grant the EBS CSI driver access too.
    kmsKeyArn: REPLACE_ME
 
api:
  config:
    # Encrypts exported project archives. Generate with: openssl rand -hex 32
    # Back up this key in your secrets manager and keep it across upgrades.
    projectPortabilityKeys: '{"active":"REPLACE_ME"}'
 
    # The OIDC application you registered against your hostname in Step 2.
    auth:
      oidc:
        # Example discovery URLs for common providers:
        #   Microsoft Entra ID: https://login.microsoftonline.com/4653d2c1-77fa-412c-bf6f-db9af1f37a97/v2.0/.well-known/openid-configuration
        #   Okta:               https://integrator-2550551.okta.com/.well-known/openid-configuration
        server: REPLACE_ME # your provider's discovery URL; the bare issuer URL also works
        clientId: REPLACE_ME
        clientSecret: REPLACE_ME
        tokenEndpointAuthMethod: client_secret_post # send the client secret in the token request body
 
# Ingress for the AWS Load Balancer Controller, with the ACM certificate
# auto-discovered from the host; adapt the class and annotations to your
# controller. 'scheme: internal' keeps the load balancer private; the health
# check must accept 200-404, and the idle timeout keeps websocket and
# streaming connections open for the length of an agent run.
ingress:
  className: alb
  host: REPLACE_ME
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/success-codes: 200-404
    alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=3600

The data root defaults to 100 GiB. Increase vfs.dataRoot.size and run a Helm upgrade to expand it without a restart. For custom storage, URLs, or ingress settings, see Advanced configuration.

Step 7: Install the application

Open the Enterprise Portal (opens in a new tab) and select the team whose license is assigned to the prod-selfhosted-v3 channel. Follow the portal's installation flow and run its Export credentials and log in commands. Use the my-values.yaml you prepared above and skip the preflight plugin step.

Install the latest available version, replacing <release-name> with the name you want to give this deployment and <your-namespace> with your application namespace:

helm install <release-name> \
  oci://registry.sema4.ai/sema4ai/prod-selfhosted-v3/blockparty \
  --values my-values.yaml \
  --namespace <your-namespace>

Omit --version to install the latest version available on the v3 channel. If you need a specific release, add --version <chart-version> using the exact chart version shown in the portal. Keep the release name, namespace, and values file for future upgrades.

Expect the VFS and sandbox Pods to sit in ContainerCreating for a few minutes on a first install: they bind the data root, and they start once the data-root workload has claimed, formatted, and mounted it.

Step 8: Point DNS at your ingress

Create the DNS record resolving your hostname to the ALB endpoint. Sign-in does not complete until the hostname resolves and TLS is serving.

Step 9: Validate

  • Check the data root: the claim is Bound and the data-root log ends with data root ready. See Verify the data root.
  • Browse to https://<hostname>. Sign-in should redirect to your identity provider.
  • Sign in with a permitted user and confirm you land in the workspace.
  • Choose the workspace's models. Agents, workflows, and Sai do not run until the workspace has a Primary and a Fast model, and every page shows the banner Choose this workspace's models until then. Click Choose models, or go to Configuration > Models, choose both models, and click Save models. The Image model is optional.

The first user of a new deployment becomes its admin. Sign in yourself first, before opening access more widely.

Step 10: Protect your keys and backups

  • Back up the database.
  • Keep the KMS key enabled and accessible for as long as you retain the database or its backups.
  • Store api.config.projectPortabilityKeys in your secrets manager so you can decrypt exported project archives.

Permanently deleting the KMS key makes stored credentials unrecoverable, even from a database backup. Disabling it prevents the application from accessing those credentials until access is restored.

Next steps

To upgrade this deployment, follow Update the application.

With the application running, continue to Administration to manage model platforms, connect integrations and data sources, and set up teams. For day-2 behavior (resizing the data root, replacing the node, upgrading the sandbox runtime), see the Operations reference.