GitOps on EKS with ArgoCD and Crossplane
This page describes how to run Datagrok on Amazon EKS, with all infrastructure and the application described in a Git repository and kept in sync by ArgoCD. It complements the other paths in Deployment: use it when you want every change, to the application or to AWS, to be a reviewed pull request.
The approach has three layers:
| Layer | Tool | Owns | Changes |
|---|---|---|---|
| Bootstrap | Terraform | VPC, EKS, and node groups, or your existing ones, plus IAM roles of the in-cluster controllers, ArgoCD, and one root Application | When a cluster is created or resized |
| Delivery | ArgoCD, in each cluster | Crossplane, External Secrets, AWS Load Balancer Controller, the Datagrok Helm chart, and optional monitoring | Every pull request to the config repo |
| AWS resources | Crossplane, in each cluster | RDS, S3, the application IAM role, and the CloudWatch log group | Every pull request to the config repo |
After the bootstrap, Terraform steps back. Releases, database and bucket changes, and new settings are commits to the config repo. Each cluster manages only its own environment, so there's no central management cluster that can change all of them.
The AWS CloudFormation (EKS) template builds a similar stack (EKS, RDS, S3, IAM roles with IRSA, a load balancer, and the Helm chart) as one CloudFormation stack that you update in the AWS console. Choose the GitOps templates on this page when every change should be a reviewed pull request, or when you run several environments from the same templates.
Both can create the VPC and the cluster or use ones you already run: see Existing VPC or EKS cluster here, and New or existing cluster for CloudFormation. If your database and storage exist too, install the Helm chart directly. To move an existing stack on ECS to EKS, see Migrate from ECS.
The argocd-crossplane folder contains a complete config repo: Terraform, the ArgoCD applications, and an environment file. The parts that every installation shares, the environment chart and the Crossplane composition, are published by Datagrok with each release, so your repo refers to them by version instead of copying them. The examples on this page are excerpts from the folder.
Published packages
Datagrok publishes three artifacts on Docker Hub for every release. They always have the same version:
| Artifact | Contents |
|---|---|
oci://registry-1.docker.io/datagrok/datagrok, tag <version>-helm | The Datagrok Helm chart. It shares the repository with the datagrok/datagrok image, and the -helm suffix keeps chart and image tags apart |
oci://registry-1.docker.io/datagrok/datagrok-env, tag <version> | The environment chart. From an environment file it creates the AWS resources, the secrets, and an ArgoCD Application that deploys the Datagrok chart of the same version |
registry-1.docker.io/datagrok/crossplane-aws, tag <version> | A Crossplane package with the DatagrokEnvironment API and the composition that turns it into AWS resources |
The environment file names one version, for example 1.28.0, and that selects all three.
Release versions of datagrok-env and crossplane-aws are published starting with the first
Datagrok release after November 2026. Until then, use 0.0.3-bleeding-edge, which is rebuilt from
the development branch every day. See Follow bleeding-edge.
Prerequisites
- An AWS account and an IAM principal that can create VPC, EKS, and IAM resources. With an existing cluster, it needs IAM and administrator access to the cluster instead of VPC and EKS rights
- Terraform 1.6+ (or OpenTofu), Helm 3.8+,
kubectl, and the AWS CLI - A Git repository for the configuration, with a read-only deploy key for ArgoCD
- An S3 bucket for the Terraform state
- An issued ACM certificate for each environment's hostname, or a wildcard certificate
Repository layout
terraform/ # bootstrap, one state per environment
main.tf variables.tf outputs.tf
argocd-values.yaml # ArgoCD settings, including the health check for the environment
env/main.tfvars # a new VPC and cluster
env/existing-vpc.tfvars.example
env/existing-cluster.tfvars.example
gitops/
bootstrap/ # app of apps, rendered with the facts Terraform passes in
platform/ # Crossplane providers and functions, with their versions
envs/
main.yaml # one file per environment
charts/datagrok-env/ # source of the published environment chart
charts/datagrok-monitoring/ # optional monitoring: stack, collector, probe, rules, dashboards
crossplane-aws/ # source of the published Crossplane package
Your config repo needs terraform/ and the first three gitops/ folders, plus
charts/datagrok-monitoring/ if you turn on monitoring. The last two are the
sources Datagrok publishes the packages from. Keep them only if you
modify the environment chart or the composition.
The templates define one environment, main. Its envs/main.yaml file is the only
environment-specific configuration. To run more environments, for example a staging copy, see
Add an environment.
How the layers connect
- Terraform creates the cluster, or reads the one you name, then installs ArgoCD and a root Application that points at
gitops/bootstrap. It passes the cluster facts (account, region, VPC, database subnet group, node security group, OIDC provider, and controller role ARNs) and the environment file as Helm values of that Application. - The bootstrap chart creates the other Applications in sync waves: the controllers first, then the
Crossplane providers, functions, and the
crossplane-awspackage, then the environment from thedatagrok-envchart. - The environment chart creates a
DatagrokEnvironmentresource. Crossplane expands it into the AWS resources through the composition. - ArgoCD waits until the
DatagrokEnvironmentreportsReady, then creates the database admin role and deploys the Datagrok chart.
No AWS credentials are stored in the cluster or in Git. Every controller, and Datagrok itself, uses IAM Roles for Service Accounts (IRSA), and RDS has IAM database authentication enabled for the Datagrok admin role. Generated passwords are kept in AWS Secrets Manager.
Bootstrap with Terraform
Terraform installs ArgoCD and creates the root Application with the cluster facts:
resource "helm_release" "root" {
name = "argocd-root"
repository = "https://argoproj.github.io/argo-helm"
chart = "argocd-apps"
namespace = "argocd"
values = [yamlencode({
applications = {
bootstrap = {
source = {
repoURL = var.config_repo_url
path = "gitops/bootstrap"
helm = {
valueFiles = ["../envs/${var.env}.yaml"]
valuesObject = {
env = var.env
aws = {
region = var.region
vpcId = local.vpc_id
dbSubnetGroup = local.db_subnet_group
oidcProvider = local.oidc_provider
crossplaneRoleArn = aws_iam_role.crossplane.arn
# ...
}
}
}
}
# ...
}
}
})]
}
The local.* facts come from the VPC and cluster Terraform creates, or from the ones you name. See
Existing VPC or EKS cluster.
The Crossplane role is limited to the resources of its own environment: the datagrok-<env> RDS
instance, bucket, log group, and application role. A provider in one environment can't change another.
Crossplane composition
The composite resource exposes only what differs between deployments. The composition turns it into about a dozen AWS resources:
db:
properties:
instanceClass: {type: string, default: db.m7g.large}
engineVersion: {type: string, default: "17.9"}
multiAz: {type: boolean, default: true}
deletionProtection: {type: boolean, default: true}
backupRetentionDays: {type: integer, default: 35}
| Composed resource | Purpose |
|---|---|
| RDS instance and security group | PostgreSQL, encrypted, Multi-AZ, IAM authentication enabled, reachable only from the cluster nodes |
| S3 bucket | Versioned, encrypted, no public access, old versions expire after storage.noncurrentVersionDays |
| IAM role and policy | Assumed by the datagrok ServiceAccount. Grants S3 on the bucket, rds-db:connect, writing to the log group, and server keys in Secrets Manager |
| CloudWatch log group | /datagrok/<env>, where Datagrok exports audit, usage, and error logs |
The bootstrap chart installs the composition as the crossplane-aws package of the environment's
version. The providers and functions it uses are installed from gitops/platform, with the versions
the package was built against.
Environment files
An environment file sets the name, hostname, and version. A release is a pull request that changes one line:
env: main
hostname: datagrok.example.com
datagrok:
version: 1.28.0
db:
host: datagrok-main.abcdefghijkl.us-east-1.rds.amazonaws.com
The rest of the file is optional. See Customize the installation.
Deploy an environment
-
Push the
terraform/andgitops/folders to your config repo and setconfig_repo_urlinterraform/env/<env>.tfvars. To use your own VPC or cluster, start from one of the.tfvars.examplefiles instead. See Existing VPC or EKS cluster. -
Create the cluster, or connect to yours, and install ArgoCD:
cd terraformterraform init -backend-config="bucket=<state-bucket>" -backend-config="key=datagrok/main.tfstate" \-backend-config="region=us-east-1"terraform apply -var-file=env/main.tfvars -
Give ArgoCD read access to the config repo:
$(terraform output -raw kubeconfig_command)kubectl -n argocd create secret generic config-repo \--from-literal=type=git --from-literal=url=git@github.com:acme/datagrok-config.git \--from-file=sshPrivateKey=./deploy-keykubectl -n argocd label secret config-repo argocd.argoproj.io/secret-type=repository -
Wait for the AWS resources. RDS takes 10 to 15 minutes on first creation:
kubectl get datagrokenvironment -
Copy the
ENDPOINTcolumn intodb.hostingitops/envs/<env>.yamland push. ArgoCD creates the database admin role and deploys Datagrok. -
Point the hostname's DNS record at the load balancer shown by
kubectl -n datagrok-<env> get ingress.
The endpoint is set by hand only once per environment. It doesn't change afterwards.
Existing VPC or EKS cluster
By default Terraform creates a VPC with public, private, and database subnets in three availability zones, and an EKS cluster in it. It can use a VPC or a cluster you already run instead. Everything after Terraform, including RDS, S3, the IAM role of Datagrok, and the Datagrok chart, works the same.
| Variable | Description |
|---|---|
existing_vpc_id | Create the cluster in this VPC |
existing_private_subnet_ids | With existing_vpc_id: subnets for the cluster nodes |
existing_cluster_name | Install into this EKS cluster. Its VPC is used too |
existing_node_security_group_id | With existing_cluster_name: the security group of the nodes, allowed to reach RDS. Default: the cluster security group |
db_subnet_ids | With either option: subnets for RDS. Terraform creates the datagrok-<env> DB subnet group from them |
install_lb_controller | false if the cluster already runs the AWS Load Balancer Controller |
With either option, vpc_cidr is ignored. With an existing cluster, so are kubernetes_version and
the node group variables. terraform destroy removes only what Terraform created, never your VPC
or cluster.
Existing VPC
env = "main"
existing_vpc_id = "vpc-0123"
existing_private_subnet_ids = ["subnet-0aaa", "subnet-0bbb", "subnet-0ccc"]
db_subnet_ids = ["subnet-0ddd", "subnet-0eee"]
config_repo_url = "git@github.com:acme/datagrok-config.git"
The VPC needs:
-
Private subnets for the nodes, in at least two availability zones, with a route to a NAT gateway. The nodes pull images from Docker Hub.
-
Private subnets for RDS in at least two availability zones. They can be the node subnets.
-
For an internet-facing load balancer, public subnets tagged
kubernetes.io/role/elb: 1, so the AWS Load Balancer Controller finds them. Or name the subnets in the environment file:gitops/envs/main.yaml (excerpt)appValues:ingress:annotations:alb.ingress.kubernetes.io/subnets: subnet-0fff,subnet-0ggg
Existing cluster
env = "main"
existing_cluster_name = "my-cluster"
db_subnet_ids = ["subnet-0aaa", "subnet-0bbb"]
install_lb_controller = false # if the cluster already runs one
config_repo_url = "git@github.com:acme/datagrok-config.git"
Terraform reads the cluster, its VPC, and its IAM OIDC provider, creates the IAM roles of the controllers, and installs ArgoCD. The cluster needs:
- An IAM OIDC provider,
which IRSA requires. Terraform reads it and stops if it doesn't exist. To create it:
eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve. - Administrator access for the principal that runs Terraform, for example an
access entry with
AmazonEKSClusterAdminPolicy, and an API endpoint reachable from where Terraform runs. - Free capacity for Datagrok, about 4 vCPU and 8 GiB, and for ArgoCD, Crossplane with its six AWS providers, and External Secrets.
- Nodes in the security group that RDS allows. Managed node groups use the cluster security group,
which is the default. For other nodes, set
existing_node_security_group_id. - No ArgoCD, Crossplane, or External Secrets of its own. The templates install all three, ArgoCD in
the
argocdnamespace. - If it runs the AWS Load Balancer Controller, a controller that handles the
albingress class, which is its default.
The VPC requirements above apply to the cluster's VPC too.
Customize the installation
Most changes are values in the environment file. They survive upgrades: a new datagrok.version
brings a new chart and composition, and your values are applied on top of them. The full list of
environment file values, with their defaults, is in
charts/datagrok-env/values.yaml.
AWS resources
| Value | Default | Description |
|---|---|---|
db.instanceClass | db.m7g.large | RDS instance class |
db.engineVersion | 17.9 | PostgreSQL version |
db.allocatedStorage, db.maxAllocatedStorage | 100, 500 | Initial storage and the storage autoscaling limit, in GB |
db.multiAz | true | Standby instance in a second availability zone |
db.backupRetentionDays | 35 | Days of automated RDS backups |
allowDataDeletion | false | true lets removal delete the database, bucket and log group. See Safeguards |
db.deletionProtection, db.skipFinalSnapshot | true, false | Keep them for production. See Safeguards |
storage.noncurrentVersionDays | 30 | Days S3 keeps overwritten and deleted file versions |
logging.retentionDays | 365 | Retention of the CloudWatch log group |
ingress.certificateArn | empty | ACM certificate for the load balancer. Empty: the controller finds an issued certificate that matches hostname |
A smaller, cheaper environment, for example for evaluation:
env: eval
hostname: datagrok-eval.example.com
datagrok:
version: 1.28.0
db:
instanceClass: db.t4g.medium
multiAz: false
backupRetentionDays: 7
Changing the instance class or storage of an existing database makes AWS modify the instance, which can restart it. Change it in a maintenance window.
Datagrok settings
Any value of the Datagrok Helm chart can be set under appValues. It's
merged over the values the environment chart computes, so it also overrides them. For example,
single sign-on, more resources for the server, and the extended database connectors:
appValues:
datagrok:
openId:
enabled: true
configEndpoint: https://login.example.com/.well-known/openid-configuration
clientId: datagrok
resources:
requests: {cpu: "4", memory: 8Gi}
limits: {cpu: "8", memory: 16Gi}
grokConnectExtended:
enabled: true
The values that connect Datagrok to the AWS resources, such as postgres.external.host,
storage.s3.bucket, and the ServiceAccount role, are computed from the environment. Override them
only if you replace the resources they point to.
Service versions
datagrok.version deploys the Datagrok Helm chart of that release, and every service runs the image
version that chart sets by default (see Images and versions). To run another
version of a service, for example a fix built for you or a connector update, pin it under images
with a tag or a sha256: digest:
datagrok:
version: 1.28.0
images:
datagrok: 1.28.1
grokConnect: 2.8.7
# A digest pins one build, also of a moving tag such as bleeding-edge
grokPipe: sha256:3f1c...
| Key | Service |
|---|---|
datagrok | Datagrok server and web application |
grokConnect | Database connectors (JDBC) |
grokConnectAdbc | Database connectors (ADBC) |
grokConnectExtended | Amazon Neptune and Cloudera Impala connectors |
grokPipe | Streaming proxy |
spawner | Starts the containers of Datagrok packages |
rabbitmq | Message queue |
A service that isn't listed follows the chart, so it moves to the new default with every release. A pinned service stays on its version until you change or remove the pin. A key that isn't in the table fails the sync with the list of valid keys.
To run a different Datagrok chart than the release, set datagrok.chartVersion, for example
1.28.1-helm.
Cluster
The cluster is set by the Terraform variables in terraform/env/<env>.tfvars and changes with
terraform apply. To use your own VPC or cluster, see
Existing VPC or EKS cluster.
| Variable | Default | Description |
|---|---|---|
region | us-east-1 | AWS region |
vpc_cidr | 10.0.0.0/16 | Address range of the new VPC. It must not overlap with other environments you connect to |
kubernetes_version | 1.33 | EKS version |
node_instance_types | ["m6i.xlarge"] | Instance types of the node group |
node_min_size, node_max_size | 2, 5 | Node group size limits |
config_repo_revision | main | Branch or tag of the config repo that ArgoCD syncs |
monitoring_role | none | none, standalone, agent, or hub. See Monitoring |
install_ebs_csi_driver | true | With monitoring_role standalone or hub: install the EBS CSI driver add-on for the Prometheus volumes. Set to false if the cluster already runs it |
Modify the environment chart or the composition
If values aren't enough, you can deploy your own copy of the environment chart, the composition, or both. With a copy, you merge the changes of each new release into it yourself, so prefer values where they work.
-
Keep
gitops/charts/datagrok-env/orgitops/crossplane-aws/, from the template folder of the release you run, in your config repo, and change them there. -
Tell the bootstrap chart to use them, in the environment file:
gitops/envs/main.yaml (excerpt)datagrok:version: 1.28.0# Deploy the environment chart from this repo instead of the published oneenvChartPath: gitops/charts/datagrok-envchartVersion: 1.28.0-helm # a chart from Git has no release default# Don't install the crossplane-aws packagecrossplanePackage: false -
With
crossplanePackage: false, movexrd.yamlandcomposition.yamlintogitops/platform/, which ArgoCD applies as they are.
A composition that creates resources with other names needs the Crossplane role policy in
terraform/main.tf updated, because the policy only allows datagrok-<env> names.
Monitoring
Monitoring is off by default. Turn it on with the Terraform variable monitoring_role and run
terraform apply. The role decides which IAM role and storage driver Terraform creates, and
Terraform passes it to ArgoCD. Everything is installed in the monitoring namespace by one chart,
datagrok-monitoring. There is one Prometheus per cluster in every role.
monitoring_role | Installs | Use it when |
|---|---|---|
none | Nothing | You don't need monitoring from this template |
standalone (recommended) | Prometheus, Alertmanager, and Grafana (kube-prometheus-stack), the collector, the probe, the RDS exporter, and the Datagrok rules and dashboards | One cluster monitors itself, in the same ArgoCD as Datagrok. The cluster must not run another Prometheus operator, because two operators conflict over the same CRDs |
agent | The collector, the probe, the RDS exporter, and a forwarder (prometheus chart with kube-state-metrics and node-exporter, two hours in memory) that sends everything to a hub with remote_write | Several Datagrok clusters report to one place. The hub stores the data, evaluates the rules, and shows the dashboards |
hub | The stack, rules, and dashboards of standalone, with the remote_write receiver on and nothing collected locally | This cluster is the place that agents report to |
To run a hub that also hosts a Datagrok environment, use standalone and turn on monitoring.ingest.
Receive from agents
A hub publishes its remote_write endpoint with monitoring.ingest. It runs a proxy that lets
through only POST /api/v1/write, with a user and password per agent:
monitoring:
ingest:
enabled: true
hostname: prometheus.example.com # agents send to https://prometheus.example.com/api/v1/write
Create the users before the first sync, one htpasswd line per agent. To cut an agent off, remove
its line:
kubectl -n monitoring create secret generic prometheus-remote-write-auth \
--from-literal=auth="$(htpasswd -nb agent1 '<password>')"
On the agent, set monitoring.forwarder.url to that address and monitoring.forwarder.username
to the user. The proxy checks who writes, not what: an agent chooses its own cluster label, so
give each agent a different cluster.
Every role that collects has three sources:
| Source | What it gives |
|---|---|
Collector (datagrok-otel) | The problems Datagrok detects by itself, its heartbeat, error counts, and reports. Datagrok pushes them as OTLP records, and the collector counts them into Prometheus series |
Probe (datagrok-probe) | The state of each service from /api/admin/health, the response time, the certificate expiry, the age of the signing key, and a real sign-in when you give it an account |
| RDS exporter (YACE) | Every CloudWatch metric of the environment's RDS instance, about 45. The exporter finds the instance by the Service=Datagrok and Environment=<env> tags that the composition sets, and reads CloudWatch through the datagrok-<env>-cloudwatch-exporter IAM role |
After the first sync, point Datagrok at the collector. Open Settings > Logger, add a
Log sync destination of type OpenTelemetry (OTLP) with the endpoint
http://datagrok-otel.monitoring.svc:4318/v1/logs. Datagrok sends its problems and the heartbeat
to it whatever levels you select. Select the error, audit, and usage levels to count errors and
reports as well. Without the destination the probe and the RDS metrics still work, and the problems
Datagrok detects don't reach Prometheus.
The same labels identify an instance everywhere, so one set of rules and dashboards serves one
cluster or many. cluster is an external label of Prometheus: alerts and the series an agent sends
to a hub carry it, and the series stored in a standalone cluster don't.
| Label | Meaning | Default in this template |
|---|---|---|
cluster | The Kubernetes cluster | datagrok-<env> |
datagrok_instance | One Datagrok instance | <env> |
tenant | Optional owner of the instance | Not set |
When a cluster runs several Datagrok instances, list them in monitoring.instances. The collector
labels each record by the namespace it comes from, datagrok-<name> unless you set namespace.
The other settings are in the environment file. The monitoring block holds the values of the
datagrok-monitoring chart, and gitops/charts/datagrok-monitoring/values.yaml lists them all:
monitoring:
stack: # standalone and hub
retention: 31d # how long Prometheus keeps metrics and consumption roll-ups
storageSize: 100Gi
grafana:
hostname: grafana.datagrok.example.com # empty: kubectl port-forward only
values: {} # merged over the kube-prometheus-stack values
forwarder: # agent
url: https://prometheus.example.com/api/v1/write # the hub's remote_write endpoint
username: datagrok # basic auth; the password is in Secrets Manager datagrok/<env>/monitoring
values: {} # merged over the prometheus chart values
cluster: datagrok-main # value of the cluster label; default datagrok-<env>
alerts: {} # RDS alert thresholds
With forwarder.username, store the password before the first sync:
aws secretsmanager create-secret --name datagrok/main/monitoring --secret-string '{"password":"<password>"}'
The forwarder scrapes the kubelets and cAdvisor, kube-state-metrics, node-exporter, and every
Service or pod annotated prometheus.io/scrape: "true".
With standalone and hub the Grafana admin password is generated in the cluster:
kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.admin-password}' | base64 -d
To let the probe sign in, create an account for it in Datagrok and store it in the cluster. The key of the JSON object is the environment name:
kubectl -n monitoring create secret generic datagrok-login-probe \
--from-literal=credentials='{"main": {"user": "probe", "password": "<password>"}}'
What it collects
| Dashboard | Shows |
|---|---|
| Datagrok / Instances | Instances answering, open problems as Datagrok reports them, the state of each service over time, response time, time since the last heartbeat, errors by package, reports, sign-in time, days until the certificate expires |
| Datagrok / RDS | CPU, connections, free storage, freeable memory, latency, IOPS, queue depth, EBS I/O burst balance, replica lag, transaction IDs used, DB load |
| Datagrok / Consumption | CPU, memory, requested CPU and memory, network, and volume usage per workload over any period within retention |
| Alerts | Fire when |
|---|---|
DatagrokHealthEndpointDown (critical), DatagrokSlowResponse, DatagrokVerySlowResponse (critical) | The instance doesn't answer, or answers slowly |
DatagrokServiceDown (critical), DatagrokPartiallyDegraded (critical), DatagrokAuxServiceDown | A service is down while the instance answers. The garbage collector, Jupyter, Grok Connect, and the spawner warn. Every other service is critical |
DatagrokProblem | Datagrok reports a problem: a failed connection or job, an error incident, slow requests, a user report. The severity is the problem's own, and the alert clears when Datagrok resolves it |
DatagrokHeartbeatMissing (critical) | Datagrok sent no heartbeat for 15 minutes |
DatagrokErrorBurst, DatagrokAutoReportBurst | Many errors from one package, or many automatic reports, in 15 minutes |
DatagrokLoginFailing (critical), DatagrokLoginSlow | The probe account can't sign in, or signing in takes over 10 seconds |
DatagrokSigningKeyStale, DatagrokSigningKeyVeryStale (critical) | The signing key wasn't rotated for 100 or 130 days |
SSLCertificateExpiry7Days, SSLCertificateExpiryToday (critical) | The certificate is about to expire |
DatagrokContainerOOMKilled, DatagrokContainerMemoryNearLimit | A Datagrok container ran out of memory, or is about to |
DatagrokRdsStorageLow (critical), DatagrokRdsCpuHigh, DatagrokRdsMemoryLow, DatagrokRdsIoBalanceLow, DatagrokRdsTransactionIdsHigh, DatagrokRdsMetricsMissing | The database runs out of a resource, or its metrics stop. The thresholds are under monitoring.alerts |
DatagrokOtelCollectorDown, DatagrokOtelRecordsRefused, DatagrokProbeDown | The collector or the probe is down, or the collector refuses records |
While an instance is unreachable, Alertmanager holds back the alerts that only restate it. It has
no receiver. Add one with monitoring.stack.values.alertmanager.config.
Consumption is recorded once an hour per workload and container. Pods are counted toward the Deployment, StatefulSet, DaemonSet, or Job that owns them, so the numbers survive pod restarts. The sum over a period is the consumption of that period:
# CPU core-hours of each workload over the last 30 days
sum by (namespace, workload) (sum_over_time(workload_container:cpu_core_hours:1h[30d]))
A cluster that already runs a Prometheus operator
Deploy the datagrok-monitoring chart by itself with stack.install: false. It then installs only
the collector, the probe, the rules, and the dashboards, into the existing namespace that you set
in namespace. If your Prometheus selects rules and ServiceMonitors by label, set
stack.objectLabels, for example {release: kube-prometheus-stack}. The chart doesn't change your
Alertmanager configuration, so the alerts that restate an unreachable instance aren't held back.
Day-to-day operations
| Task | How |
|---|---|
| Release to an environment | Pull request that changes datagrok.version in envs/<env>.yaml |
| Roll back | Revert the pull request |
| Follow the development builds | See Follow bleeding-edge |
| Change RDS or S3 settings | Pull request to the environment file |
| Preview a change | terraform plan for the cluster. argocd app diff and crossplane render for Applications and AWS resources |
Follow bleeding-edge
bleeding-edge is the current state of Datagrok development. Datagrok builds it from the master
branch every night, sometimes also during the day, and publishes everything under moving tags: the bleeding-edge service images, the Datagrok chart 0.0.3-bleeding-edge-helm, and the
datagrok-env and crossplane-aws packages 0.0.3-bleeding-edge (see
Tag conventions). It isn't a tested release. Use it for a development
or evaluation environment that tries new features before they ship, not for production: database
migrations only run forward, so an environment that ran bleeding-edge can't go back to an older
release.
Set the version in the environment file:
datagrok:
version: 0.0.3-bleeding-edge
A moving tag doesn't redeploy anything by itself. Crossplane checks the crossplane-aws package for a
new build, but ArgoCD treats a chart version as fixed and can serve a cached copy for up to a day, and
running pods keep the image they started with. To get each night's build, refresh the environment
daily with a scheduled job that resolves the moving image tags to digests and commits them under
images. Every refresh is then a commit that ArgoCD rolls out, and that you can revert. For example,
with GitHub Actions in the config repo:
name: Refresh bleeding-edge
on:
schedule:
- cron: "0 7 * * *" # after the nightly build
workflow_dispatch:
permissions:
contents: write
jobs:
refresh:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Pin the latest bleeding-edge images
run: |
file=gitops/envs/dev.yaml
for pair in datagrok:datagrok grokPipe:grok_pipe spawner:grok_spawner grokConnectAdbc:grok_connect_adbc; do
digest=$(docker buildx imagetools inspect "datagrok/${pair#*:}:bleeding-edge" --format '{{.Manifest.Digest}}')
yq -i ".images.${pair%%:*} = \"$digest\"" "$file"
done
git diff --quiet && exit 0
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git commit -am "Refresh bleeding-edge images" && git push
The job pins the services whose chart default is bleeding-edge. Services that the chart pins to a
version, such as grokConnect, follow the chart. After a refresh that changes the chart or the
composition, run argocd app get datagrok-<env> --hard-refresh to pick up the new build right away.
Safeguards
Crossplane has no plan step that asks AWS what a change will do. A preview shows which fields change, not whether AWS modifies, restarts, or replaces a resource. The templates protect data against that:
- Removing the environment never deletes its data. The RDS instance, the S3 bucket and the log group
are created with
deletionPolicy: Orphan, so deleting theDatagrokEnvironment, thedatagrok-<env>application, the crossplane-aws package or its XRD leaves them in AWS. Restoring the environment with the sameenvadopts them again. allowDataDeletionis the safety switch. Until it istrueon the running environment, Kubernetes rejects any change that turns offdb.deletionProtection, turns ondb.skipFinalSnapshotorstorage.forceDestroy, or lowersdb.backupRetentionDaysorstorage.noncurrentVersionDays. Set it in its own commit and sync it before you lower a guard: a commit that does both is rejected. Theenvof an environment can't be changed.- Behind the switch, RDS has deletion protection and takes a final snapshot, automated backups are kept for their retention period after the instance is deleted, and a non-empty bucket can't be deleted.
- Generated passwords are created once and kept in Secrets Manager under
datagrok/<env>. Neither ArgoCD nor a resync regenerates them. - Every version is pinned: the Datagrok release by
datagrok.version, and the providers, functions, and controllers ingitops/platformandgitops/bootstrap. If you run a second environment, upgrade it first.
Add an environment
Each environment gets its own cluster, ArgoCD, and AWS resources, built from the same templates:
- Copy
terraform/env/main.tfvarstoterraform/env/<env>.tfvars. Setenvand a VPC CIDR that doesn't overlap with other environments, or another existing VPC or cluster. A cluster holds one environment, because each environment installs its own ArgoCD. - Copy
gitops/envs/main.yamltogitops/envs/<env>.yaml. Setenv,hostname, and the version. - Run the deploy steps with the new file and a new state key, for example
key=datagrok/<env>.tfstate.
Each environment has its own datagrok.version, so you can release to one environment first.
Troubleshooting
| Symptom | Check |
|---|---|
platform Application is OutOfSync on first deploy | Normal for a few minutes. Crossplane installs CRDs asynchronously, and the Application retries until they exist |
DatagrokEnvironment kind is unknown | kubectl get configuration datagrok-aws shows whether the crossplane-aws package installed. A version that isn't published fails to pull |
DatagrokEnvironment stays not Ready | kubectl describe datagrokenvironment and kubectl get managed show the AWS error for each resource |
AWS API returns AccessDenied | The Crossplane role policy in terraform/main.tf is scoped to datagrok-<env> names. A renamed resource needs the policy updated |
| Datagrok can't connect to the database | db.host in the environment file must match the RDS endpoint, and the datagrok-db-bootstrap job must have completed. In an existing cluster, the nodes must be in the security group RDS allows: set existing_node_security_group_id |
terraform apply can't find an OIDC provider | The existing cluster has no IAM OIDC provider. Create it with eksctl utils associate-iam-oidc-provider |
terraform apply is denied by Kubernetes | The principal running Terraform has no administrator access to the existing cluster. Add an access entry |
A monitoring Prometheus or Alertmanager pod stays Pending | Its volume isn't provisioned: the EBS CSI driver is missing. Set install_ebs_csi_driver = true, or install the add-on yourself |
DatagrokRdsMetricsMissing fires | kubectl -n monitoring logs deploy/yace shows the CloudWatch or IAM error |
| The ingress gets no load balancer | The AWS Load Balancer Controller finds no public subnets. Tag them kubernetes.io/role/elb: 1, or set alb.ingress.kubernetes.io/subnets |
See also
- Install Datagrok with Helm: the chart this page deploys, and all its values
- AWS CloudFormation (EKS): EKS, RDS, and S3 as one CloudFormation stack, also on an existing cluster
- AWS CloudFormation (ECS): for existing ECS stacks
- Images and versions: image tags, chart tags, and the default version of each service
- Upgrade