Version 3.0
AWS EKS

Deploy on AWS EKS

Install a self-hosted deployment into your own AWS account, on an EKS cluster with a sandbox-capable node.

Audience: IT (the enabler)

This guide assumes you are comfortable operating EKS. It does not cover general Kubernetes administration.

Before you start, read Cluster and node requirements. The cluster this guide builds is not a generic one, and its storage and runtime prerequisites are the part most likely to catch you out.

Part 1: Pre-deployment bill of materials

Hand this to procurement and your cloud and security teams on day 0.

Decide these carefully, they are painful to retrofit:

  • Hostname. TLS, the OIDC redirect URI, and ingress all derive from it. Changing it later means re-issuing certificates and reconfiguring your identity provider.
  • Region. Moving regions later means a rebuild.

Components

CategoryComponentSpecNotes
ComputeEKS clusterEKS 1.36 or newer, with the cluster add-ons the platform depends on, including the EBS CSI driverSee Cluster and node requirements
ComputeApplication nodeA single x86_64 m8i.8xlarge (32 vCPU, 128 GiB) with nested virtualization explicitly enabled, in a node group of one. The instance type we primarily test against, and thus supportHosts the application and every sandbox run. Not interchangeable with a general-purpose node group. See Node sizing
StorageNode root volumeEncrypted gp3Sized for the node image and container images
StorageData root StorageClassCreated by the application chart on the EBS CSI driver: gp3, encrypted with your KMS key, WaitForFirstConsumer, expandable, reclaimPolicy: DeleteNothing to create. Overridable with a custom class when you need tuned IOPS or throughput; see Advanced configuration
StorageData root volumeA gp3 EBS volume, 100 GiB by default, that a Pod of the application claims from that StorageClass on the nodeA cache for materialized workspaces and sandbox images, not a system of record. Deleted with the node; a replacement starts cold. See The data root
NetworkVPC and subnetsPrivate subnets across multiple availability zones, with NAT for egress. The application node group can span any of them
NetworkCluster ingressAn ALB fronting the application, with an issued ACM certificate in the same regionYou operate the ingress and terminate TLS there
NetworkDNS recordYour chosen hostname resolving to the ingress endpoint
DataPostgreSQLAurora PostgreSQL Serverless v2, PostgreSQL 17 or newer, private onlyPrimary application data store
DataS3 bucketA bucket for the deployment, with a customer-managed KMS keyDurable storage for files, artifacts, and the workspace filesystem
DataKMS keyCustomer-managed, with rotation enabledEnvelope encryption of secrets at rest, and encryption of the data root volume
IdentityOIDC IdP appAn application registered in Entra ID, Okta, or Auth0See Identity providers (OIDC)
IdentityService account for AWS accessA Kubernetes service account in the application namespace, bound through EKS Pod Identity to an IAM role that can use the bucket and the KMS keyCreate it before the install and name it in the values file in Step 6
IdentityEBS CSI driver key accessThe IAM identity the EBS CSI driver runs as, allowed to use the KMS keyThe chart-created StorageClass encrypts with your KMS key; without the grant the data root volume never provisions
AccessAWS and EKS admin accessPermissions for the deploying administrator, plus kubectl and helm on their workstationEKS access entries must reference stable IAM role or user ARNs
EgressOutbound HTTPSUnrestricted egress recommendedNeeded for the Sema4.ai services, the sandbox runtime's chart and image, and everything you connect the platform to. See Network endpoints

Node sizing

The application node is a single m8i.8xlarge: 32 vCPU and 128 GiB. It is the instance type we primarily test against, and thus support. It carries the whole platform and every concurrent sandbox run, with headroom for several teams sharing the platform; see Allocatable capacity for what actually reaches the platform.

This release does not scale horizontally, so the node is the ceiling on concurrent agent work for the life of the deployment. Replacing the node costs downtime and a cold cache rather than data. See Caveats and limits.

Part 2: Deployment steps

Step 1: Pick the hostname

Choose the hostname on a domain you control, for example finance-agents.company.com. Whether it resolves publicly or only inside your network is your choice. TLS, the OIDC redirect URI, and the ingress configuration all derive from it. You point DNS at the ingress in Step 8.

Issue an ACM certificate for the hostname in the same region as the cluster, and confirm it reaches Issued before you provision.

Step 2: Register the OIDC application

Register an OIDC application in your identity provider and capture the Discovery URL, Client ID, and Client Secret.

Configure it with:

  • Callback URL: https://<hostname>/api/v1/auth/callback
  • Logout URL: https://<hostname>
  • Application login URI: https://<hostname>/login
  • Allowed web origin: https://<hostname>
  • Scopes: openid, profile, email

Follow the guide for your provider: Microsoft Entra ID · Auth0 · Okta.

Step 3: Provision AWS infrastructure

Provision the infrastructure using the bill of materials as the spec. The end state must include:

  • An EKS cluster with the required add-ons, including the EBS CSI driver, and kubectl access configured against it.
  • The application node, with nested virtualization enabled, in a node group of one.
  • Aurora PostgreSQL, version 17 or newer, reachable from the cluster.
  • An S3 bucket and a KMS key.
  • The application namespace, a Kubernetes service account in it, and an IAM role that can use the bucket and the key, bound to the account through EKS Pod Identity. You name the account in the values file in Step 6.
  • An ALB ingress and the ACM certificate from Step 1.

Node bootstrap needs nothing platform-specific: no user data, no script, no packages, and no data volume. The nodes run the stock EKS AL2023 image, and nested virtualization comes from the instance type you provision. Bootstrap does not install the sandbox runtime; that is Step 4, and it prepares nodes that join later on its own.

There is no StorageClass to create: the chart creates its own for the data root — gp3 on the EBS CSI driver, encrypted with your KMS key, meeting the contract in Storage for the data root. If you need tuned IOPS or throughput, operate your own class instead; see Advanced configuration.

The KMS key needs a grant for the EBS CSI driver. The chart-created StorageClass encrypts the data root with your key, the managed AmazonEBSCSIDriverPolicy grants no KMS actions, and without them the data root claim stays Pending with a misleading InvalidVolume.NotFound error. Grant the driver's IAM identity use of the key before you install; the exact policy is in the Operations reference.

Then confirm the node and the storage prerequisites:

  • The node is Ready.
  • /dev/kvm is present on the node.
  • kubectl get csidriver ebs.csi.aws.com returns the driver.

Step 4: Install the sandbox runtime

Install Kata Containers into the cluster, once, following Install the sandbox runtime. Verify it before continuing:

kubectl get runtimeclass kata-clh
kubectl get nodes -l katacontainers.io/kata-runtime=true

Nothing checks this at install time. The application may install without the sandbox runtime, but it will not operate correctly: agent runs fail, and the failure does not point back at the missing runtime. Make sure both commands above pass before you proceed.

Step 5: Create the database and its roles

The application does not create its own database, and it connects with three least-privilege roles rather than the Aurora master user. Connect to the cluster as the master user and run the following, with your own names and strong passwords:

CREATE ROLE <migrator_role> LOGIN PASSWORD '<migrator_password>'
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;
CREATE ROLE <definer_role> NOLOGIN
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION BYPASSRLS;
CREATE ROLE <app_role> LOGIN PASSWORD '<app_password>'
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;
GRANT <definer_role> TO <migrator_role>;
CREATE DATABASE <database>;
GRANT <migrator_role> TO CURRENT_USER;
ALTER DATABASE <database> OWNER TO <migrator_role>;
REVOKE <migrator_role> FROM CURRENT_USER;
GRANT CONNECT ON DATABASE <database> TO <app_role>, <definer_role>;

The migrator role applies the schema on install and upgrade, the app role is what the application connects as at runtime, and the definer role owns privileged functions and cannot log in; see The database roles for the full rationale.

Capture the database name, the three role names, and the two passwords: they go into the values file.

Step 6: Prepare your values file

The values file is your readiness check. Complete it before you install. Every value in it is something a prerequisite from Part 1 produced, so a value you cannot fill in is a prerequisite that is not ready. Do not start Step 7 with a REPLACE_ME left in the file.

Start from the template below and replace every REPLACE_ME. Only what the chart cannot know about your environment is in it; everything else is a chart default, tuned for the supported node. Keep the file: you reuse it for every upgrade.

Download my-values-v3-aws-eks.example.yaml
# Helm values for a self-hosted deployment on AWS EKS.
#
# Complete every REPLACE_ME before you install; a value you cannot fill in is
# a prerequisite that is not ready yet. Keep this file: you reuse it for
# upgrades. Everything not set here is a chart default, tuned for the
# supported 32 vCPU / 128 GiB node.
 
# The Kubernetes service account that gives the application access to AWS
# resources. Create it in the application namespace and bind it through EKS
# Pod Identity to the IAM role for the bucket and the KMS key.
serviceAccount:
  create: false
  name: REPLACE_ME
 
# The application database and its three roles, created before the install
# (see "Create the database and its roles" in the EKS guide).
postgres:
  host: REPLACE_ME
  database: REPLACE_ME
  appRole: REPLACE_ME
  appPassword: REPLACE_ME
  definerRole: REPLACE_ME
  migratorRole: REPLACE_ME
  migratorPassword: REPLACE_ME
 
# The AWS resources from the bill of materials, in one block. From these the
# chart derives everything AWS-shaped:
#   - object storage at s3://<bucket>[/<prefix>], reached through the service
#     account's IAM role (no static keys);
#   - a gp3 StorageClass on the EBS CSI driver, encrypted with your KMS key,
#     and the data root claimed from it as a raw block volume on the node
#     (100 GiB by default; raise vfs.dataRoot.size later and upgrade, the
#     filesystem grows without a restart);
#   - the AWS region for SDK calls (the IAM role supplies credentials, not a
#     region);
#   - the sandbox wiring for EKS: the Kata runtime you installed before the
#     platform, reached through the node's containerd, with the release
#     installing no Kata of its own.
# Anything set explicitly elsewhere in this file wins over a derived value.
infrastructure:
  platform: aws
  aws:
    region: REPLACE_ME
    # The bucket name only. The optional key prefix keeps several
    # deployments apart in one bucket.
    s3BucketName: REPLACE_ME
    # s3KeyPrefix: ""
    # Full ARN of the KMS key that encrypts the data root volumes. The EBS
    # CSI driver's IAM identity must be granted use of this key (see the KMS
    # grant callout in the EKS guide) or the volume never provisions.
    kmsKeyArn: REPLACE_ME
 
api:
  config:
    # Encryption keys, yours to generate and to keep. Generate a long random
    # string for each, for example with `openssl rand -hex 32`.
    # secretsKeys encrypts the credentials the platform stores in its
    # database: model platform keys, integration and OAuth tokens. The
    # database outlives the cluster, and without the exact key its encrypted
    # contents cannot be decrypted — the database may become unusable.
    # projectPortabilityKeys protects exported project archives the same
    # way. Keep this file backed up somewhere safe outside the cluster.
    secretsKeys: '{"v1":"REPLACE_ME"}'
    projectPortabilityKeys: '{"active":"REPLACE_ME"}'
 
    # Your hostname as https://<hostname>, and the OIDC application you
    # registered against it.
    auth:
      allowedOrigins: '["https://REPLACE_ME"]'
      webAppUrl: https://REPLACE_ME
      oidc:
        server: REPLACE_ME # your provider's discovery URL; the bare issuer URL also works
        clientId: REPLACE_ME
        clientSecret: REPLACE_ME
        redirectUri: https://REPLACE_ME/api/v1/auth/callback
    realtime:
      webhookPublicBaseUrl: https://REPLACE_ME
 
# Ingress for the AWS Load Balancer Controller, with the ACM certificate
# auto-discovered from the host; adapt the class and annotations to your
# controller. 'scheme: internal' keeps the load balancer private; the health
# check must accept 200-404, and the idle timeout keeps websocket and
# streaming connections open for the length of an agent run.
ingress:
  className: alb
  host: REPLACE_ME
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/success-codes: 200-404
    alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=3600

Three things are worth a moment. The data root size is not in the file because 100 GiB is the chart default; you can raise it later (vfs.dataRoot.size) without a restart, so start small. Every value the infrastructure block derives has an explicit override that wins over the convention — a StorageClass of your own, or vfs.blobs.uri set directly — but the file above is the supported shape; see Advanced configuration. The data root mounts at /var/lib/blockparty/<release>, named after the Helm release you install under; nothing in the file needs to say so. And no internal service tokens are in the file: the chart generates them at first install, reuses them across upgrades, and they die with the deployment. The two encryption keys are in the file, deliberately — they protect what the platform stores in your database, so their custody stays with you and this file, not with the cluster.

Step 7: Install the application

The self-hosted release channel for version 3 is not open yet. There is no install command to publish. This step will carry the registry login and the chart install once the channel ships.

Expect the VFS and sandbox Pods to sit in ContainerCreating for a few minutes on a first install: they bind the data root, and they start once the data-root workload has claimed, formatted, and mounted it.

Step 8: Point DNS at your ingress

Create the DNS record resolving your hostname to the ALB endpoint. Sign-in does not complete until the hostname resolves and TLS is serving.

Step 9: Validate

  • Check the data root: the claim is Bound and the data-root log ends with data root ready. See Verify the data root.
  • Browse to https://<hostname>. Sign-in should redirect to your identity provider.
  • Sign in with a permitted user and confirm you land in the workspace.
  • Run a smoke test that exercises a sandboxed run, which confirms the data root, the sandbox runtime, the database, and object storage together. Expect the first run to be slower than steady state while the caches fill.

The first user of a new deployment becomes its owner. Sign in yourself first, before opening access more widely.

Step 10: Back up the encryption keys

The two encryption keys you generated into the values file encrypt the secrets the platform stores in its database — model platform credentials, integration and OAuth tokens — and the database outlives the cluster.

Anything encrypted with a lost key cannot be read back. Stored credentials and connections cannot be recovered, and the database may become unusable. Treat the keys as irreplaceable.

Store the two key values in your secrets manager, alongside your other infrastructure credentials, so they survive the loss of the machine that holds the values file. Treat them as root credentials.

Next steps

With the application running, continue to Administration to configure models, connect integrations and data sources, and set up teams. For day-2 behavior — resizing the data root, replacing the node, upgrading the sandbox runtime — see the Operations reference.

What this article will cover

  • The full bill of materials with concrete sizes, versions, and SKUs
  • The install commands, once the self-hosted channel ships
  • Screenshots of the install flow
  • Upgrading an existing deployment
  • Teardown, and what survives it