openwork/docs/gcp-gke-helm.md

Deploy OpenWork EE on Google Cloud with GKE and Helm

Status: self-host operator guide Related: packaging/helm/openwork-ee, packaging/helm/openwork-ee/examples/values.gcp-ingress.yaml

This is the recommended Google Cloud path for a first production-like OpenWork EE self-host install. Use Helm on GKE Autopilot with Cloud SQL for MySQL. For web/API exposure, use GKE Ingress with Google-managed certificates, a reserved global IP address, and explicit backend health checks.

Google recommends Gateway API for new L7 traffic management, and GKE Ingress is in maintenance mode. The current OpenWork chart emits Ingress resources, so GKE Ingress is the simplest supported GCP path today. Treat Gateway API support as a future chart/platform hardening item.

Do not use raw Kubernetes LoadBalancer Services as the normal GCP path for OpenWork. GKE LoadBalancer Services are useful for TCP services and quick smoke tests, but the customer-facing web app and SSO flow need HTTP(S) load balancing, host routing, managed certificates, and backend health checks.

What this deploys

• Den API on port 8788
• Den Web on port 3005
• optional inference service, disabled by default
• one Cloud SQL for MySQL database
• one single-org OpenWork deployment
• one external GKE Ingress backed by a Google Cloud Application Load Balancer
• one Google-managed certificate covering web and API hosts

Google Cloud owns the GKE cluster, Autopilot compute lifecycle, VPC networking, Cloud Load Balancing, managed certificates, Cloud SQL, IAM, and firewall rules. The OpenWork Helm chart owns OpenWork Deployments, Services, ConfigMaps, Secrets, health probes, the optional Ingress, and the database migration Job. The BackendConfig and ManagedCertificate resources in this guide are GKE-specific platform resources applied alongside the chart.

Use Helm or something else?

Use Helm on GKE for Google Cloud unless the customer explicitly cannot run Kubernetes. The OpenWork EE release artifact is already a Helm chart, and GKE Autopilot keeps the first customer path small while still supporting migration Jobs, separate web/API services, SSO-ready HTTPS, and later enterprise network controls. The practical gap to fill is GCP-specific ingress and database guidance, not a different OpenWork packaging format.

Prerequisites

• Google Cloud CLI authenticated to the target project.
• kubectl, helm, and gke-gcloud-auth-plugin.
• Permission to create GKE, Compute Engine networking and global addresses, Cloud SQL, Service Networking, DNS, IAM, and Kubernetes resources.
• Enabled APIs: Kubernetes Engine API, Compute Engine API, Cloud SQL Admin API, and Service Networking API.
• A real admin email address for the first owner account.
• A domain you control, such as openwork.example.com and api.openwork.example.com.

Google Cloud docs used for this guide:

• GKE Autopilot clusters: https://cloud.google.com/kubernetes-engine/docs/how-to/creating-an-autopilot-cluster
• GKE Ingress for Application Load Balancers: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
• GKE external Ingress and NEGs: https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing
• GKE managed certificates: https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs
• GKE Ingress configuration and BackendConfig: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-configuration
• GKE Ingress health checks: https://cloud.google.com/kubernetes-engine/docs/troubleshooting/ingress-health-checks
• Cloud SQL private IP: https://cloud.google.com/sql/docs/mysql/private-ip
• Cloud SQL from GKE: https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine

1. Create the GKE cluster

For a first deployment, create a regional Autopilot cluster:

export GCP_PROJECT=REPLACE_PROJECT_ID
export GCP_REGION=us-central1
export GKE_CLUSTER=openwork-ee

gcloud config set project "$GCP_PROJECT"

gcloud services enable \
  container.googleapis.com \
  compute.googleapis.com \
  sqladmin.googleapis.com \
  servicenetworking.googleapis.com

gcloud container clusters create-auto "$GKE_CLUSTER" \
  --location "$GCP_REGION" \
  --project "$GCP_PROJECT"

# If the cluster command reports that the auth plugin is missing:
gcloud components install gke-gcloud-auth-plugin --quiet

gcloud container clusters get-credentials "$GKE_CLUSTER" \
  --location "$GCP_REGION" \
  --project "$GCP_PROJECT"

kubectl get nodes

GKE enables HTTP load balancing by default. Do not disable it; GKE Ingress needs that add-on.

Some standalone Google Cloud SDK installations do not include gke-gcloud-auth-plugin. If kubectl cannot authenticate after cluster creation, install the component as shown above, rerun get-credentials, and verify with kubectl get nodes before debugging Kubernetes resources.

2. Create Cloud SQL for MySQL

Create Cloud SQL for MySQL with private IP in the same VPC as the GKE cluster. The most important requirements are:

• MySQL 8-compatible Cloud SQL instance.
• Database name: openwork_den.
• Private services access configured for the VPC.
• Private IP enabled on the Cloud SQL instance.
• GKE is VPC-native and can reach the private IP.
• Backups enabled.
• Public IP disabled for production unless there is a documented exception.

Private IP requires a one-time private services access connection for the VPC:

export VPC_NETWORK=default
export SQL_RANGE=openwork-sql-range

gcloud compute addresses create "$SQL_RANGE" \
  --global \
  --purpose=VPC_PEERING \
  --prefix-length=16 \
  --network="$VPC_NETWORK"

gcloud services vpc-peerings connect \
  --service=servicenetworking.googleapis.com \
  --ranges="$SQL_RANGE" \
  --network="$VPC_NETWORK"

Create the instance and database:

export SQL_INSTANCE=openwork-ee-mysql

gcloud sql instances create "$SQL_INSTANCE" \
  --database-version=MYSQL_8_0 \
  --region="$GCP_REGION" \
  --network="projects/$GCP_PROJECT/global/networks/$VPC_NETWORK" \
  --no-assign-ip

gcloud sql databases create openwork_den \
  --instance="$SQL_INSTANCE"

gcloud sql users create openwork \
  --instance="$SQL_INSTANCE" \
  --password=REPLACE_DB_PASSWORD

Get the private IP:

gcloud sql instances describe "$SQL_INSTANCE" \
  --format='value(ipAddresses[0].ipAddress)'

Example database URL:

mysql://openwork:<password>@<cloud-sql-private-ip>:3306/openwork_den

This guide uses direct private IP because the current OpenWork chart does not inject Cloud SQL Auth Proxy sidecars. Cloud SQL Auth Proxy is a stronger future hardening path when the chart supports sidecars or an operator-managed proxy pattern.

If the Cloud SQL instance enforces encrypted client connections, use ?sslaccept=accept for the simple private-MySQL smoke path. This keeps TLS on without requiring a cloud CA bundle to be mounted into the OpenWork image. Use strict certificate verification later, after you provide the required CA bundle, with a hardened value such as sslmode=verify-ca or sslmode=verify-full. Verify the same URL works for both the migration Job and runtime pods before testing the browser flow.

Before installing OpenWork, verify network access from the cluster:

kubectl run mysql-client \
  --rm \
  -it \
  --restart=Never \
  --image=mysql:8 \
  -- mysql \
    --host="REPLACE_CLOUD_SQL_PRIVATE_IP" \
    --user=openwork \
    --password \
    --execute "select 1"

3. Reserve a global IP and create GKE resources

Reserve a global IP address for the HTTPS load balancer:

gcloud compute addresses create openwork-ee-ip \
  --global

gcloud compute addresses describe openwork-ee-ip \
  --global \
  --format='value(address)'

Create the namespace:

kubectl create namespace openwork-ee

Create a Google-managed certificate resource:

kubectl apply -n openwork-ee -f - <<'YAML'
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: openwork-ee-cert
spec:
  domains:
    - REPLACE_WEB_HOST
    - REPLACE_API_HOST
YAML

Create explicit backend health checks for the two OpenWork services:

kubectl apply -n openwork-ee -f - <<'YAML'
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: openwork-ee-den-api-backend
spec:
  healthCheck:
    type: HTTP
    requestPath: /ready
    port: 8788
    checkIntervalSec: 15
    timeoutSec: 5
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: openwork-ee-den-web-backend
spec:
  healthCheck:
    type: HTTP
    requestPath: /api/ready
    port: 3005
    checkIntervalSec: 15
    timeoutSec: 5
YAML

The Helm values annotate the OpenWork Services so GKE associates these BackendConfig objects with the Google Cloud backend services.

4. Prepare Helm values

Copy the starter file:

cp packaging/helm/openwork-ee/examples/values.gcp-ingress.yaml values.gcp.yaml

Replace every REPLACE_* placeholder.

Generate secrets:

openssl rand -base64 48
openssl rand -base64 48

Use the first value for secret.values.betterAuthSecret and the second for secret.values.denDbEncryptionKey. Do not reuse either value across environments.

To send transactional email, configure SMTP in the same values file:

secret:
  values:
    emailFrom: "OpenWork <no-reply@example.com>"
    smtpHost: "smtp.example.com"
    smtpPort: "587"
    smtpUser: "openwork@example.com"
    smtpPass: "REPLACE_SMTP_PASSWORD"
    smtpSecure: "false"

These values become EMAIL_FROM, SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, and SMTP_SECURE in Den API. If you use secret.create=false, add those keys to the existing Kubernetes Secret referenced by secret.existingSecret. SMTP delivery requires both EMAIL_FROM and SMTP_HOST; leave smtpHost blank only when SMTP-backed transactional email should be disabled.

Use a values file, not a long list of --set flags. Several OpenWork values are comma-separated strings, such as config.public.corsOrigins, and plain --set parsing commonly breaks them.

Make sure your file uses the current chart keys. Public URL values belong under config.public.*; database and app secrets belong under secret.values.*. Values such as config.urls, config.databaseUrl, or secrets.* are ignored by the chart.

Before installing, render the chart and verify the migration Job will use your Cloud SQL URL:

helm template openwork-ee oci://ghcr.io/different-ai/charts/openwork-ee \
  --version REPLACE_OPENWORK_VERSION \
  --namespace openwork-ee \
  -f values.gcp.yaml > /tmp/openwork-rendered.yaml

grep -E 'DATABASE_URL|DEN_BASE_URL|DEN_WEB_PUBLIC_ORIGIN|EMAIL_FROM|SMTP_HOST|SMTP_PORT|SMTP_SECURE' /tmp/openwork-rendered.yaml

Redact secrets before sharing rendered manifests or terminal output.

5. Install OpenWork

Published chart releases live in GHCR:

helm upgrade --install openwork-ee oci://ghcr.io/different-ai/charts/openwork-ee \
  --version REPLACE_OPENWORK_VERSION \
  --namespace openwork-ee \
  --create-namespace \
  -f values.gcp.yaml

For a checkout-local test:

helm upgrade --install openwork-ee ./packaging/helm/openwork-ee \
  --namespace openwork-ee \
  --create-namespace \
  -f values.gcp.yaml

If GHCR image pulls fail with ImagePullBackOff, authenticate to GHCR and add an imagePullSecrets entry. Public releases should not require this, but private packages or private forks do:

kubectl create secret docker-registry ghcr-pull-secret \
  --namespace openwork-ee \
  --docker-server=ghcr.io \
  --docker-username="$GITHUB_USER" \
  --docker-password="$GITHUB_TOKEN"

imagePullSecrets:
  - name: ghcr-pull-secret

6. Migration troubleshooting

The migration Job runs before the Deployments are useful. If it fails, fix that before debugging web/API readiness.

Avoid kubectl describe job openwork-ee-migrate in shared reports because the hook Job currently includes DATABASE_URL and DEN_DB_ENCRYPTION_KEY in the rendered environment. Use logs and redacted rendered manifests instead.

For a retained-log debug attempt, temporarily disable hook behavior:

migrations:
  enabled: true
  hook: false
  backoffLimit: 0

Then run Helm and inspect the normal Job logs:

helm upgrade --install openwork-ee oci://ghcr.io/different-ai/charts/openwork-ee \
  --version REPLACE_OPENWORK_VERSION \
  --namespace openwork-ee \
  --create-namespace \
  -f values.gcp.yaml \
  --wait=false

kubectl get jobs,pods -n openwork-ee
kubectl logs -n openwork-ee -l job-name=openwork-ee-migrate --all-containers=true

Return to the default hook mode after debugging:

migrations:
  enabled: true
  hook: true
  backoffLimit: 2

7. Point DNS at the global load balancer IP

Get the reserved IP address:

gcloud compute addresses describe openwork-ee-ip \
  --global \
  --format='value(address)'

Create DNS records:

• openwork.example.com -> the reserved global IP address.
• api.openwork.example.com -> the reserved global IP address.

GKE can take several minutes to provision the load balancer. Google-managed certificates can take up to an hour to become active after DNS points at the load balancer.

Check status:

kubectl get ingress -n openwork-ee
kubectl describe managedcertificate openwork-ee-cert -n openwork-ee
kubectl describe ingress openwork-ee -n openwork-ee

If you are still using temporary hosts before DNS/TLS is ready, temporarily update the corresponding config.public.* origins in values.gcp.yaml, then run helm upgrade again. Do not leave production deployments on raw IPs or placeholder hostnames.

The current chart rolls the Den API, Den Web, and inference pods automatically when ConfigMap or Secret content changes. On older chart versions, manually restart the deployments after changing public origin values:

kubectl rollout restart deployment/openwork-ee-den-api deployment/openwork-ee-den-web -n openwork-ee
kubectl rollout status deployment/openwork-ee-den-api -n openwork-ee --timeout=180s
kubectl rollout status deployment/openwork-ee-den-web -n openwork-ee --timeout=180s

8. Verify readiness

Check Kubernetes state:

helm status openwork-ee -n openwork-ee
kubectl get pods -n openwork-ee
kubectl get jobs -n openwork-ee
kubectl get ingress -n openwork-ee
kubectl describe backendconfig openwork-ee-den-api-backend -n openwork-ee
kubectl describe backendconfig openwork-ee-den-web-backend -n openwork-ee
kubectl logs -n openwork-ee deploy/openwork-ee-den-api
kubectl logs -n openwork-ee deploy/openwork-ee-den-web

Check readiness from your machine:

curl -fsS https://api.openwork.example.com/ready
curl -fsS https://openwork.example.com/api/ready

9. Bootstrap the first owner

The chart defaults to single_org. Set these before first sign-in:

config:
  tenancy:
    mode: "single_org"
    singleOrgName: "Acme"
    singleOrgSlug: "acme"
    ownerEmails: "admin@acme.com"
    requireEmailVerification: "false"
  public:
    bootstrapAdminEmails: "admin@acme.com"
secret:
  values:
    initialAdminBootstrapCode: "REPLACE_BOOTSTRAP_CODE"

For releases that include initial-administrator bootstrap, inject the release-documented one-time setup secret through the Kubernetes Secret referenced by secret.existingSecret. Do not store the code in the values file or a ConfigMap. Then open https://openwork.example.com/setup, enter the configured owner email and one-time operator code, and create the first account. OpenWork creates the singleton organization, grants owner and configured platform-admin access, and signs the administrator in. Public signup remains disabled. After the first user exists, the setup code cannot bootstrap another account.

ownerEmails and bootstrapAdminEmails authorize roles; neither setting creates an account or password. There is no default administrator password. Chart versions without /setup do not support private initial-administrator bootstrap and must be upgraded before following this step.

10. Configure SSO with a test IdP

Self-hosted installs keep plan gating off unless the operator explicitly sets DEN_PLAN_GATING_ENABLED=true, so SSO management should be available in the EE self-host default.

Use OIDC first for a smoke test because most IdPs provide discovery metadata. Auth0, Okta trial, and Google Cloud Identity test tenants all work as realistic demo IdPs.

Configure the IdP application with this callback URL:

https://openwork.example.com/api/auth/sso/callback/openwork-sso-<org-id>

In OpenWork, sign in as the owner, open the organization SSO settings, and enter the IdP issuer/client details. After saving, the organization sign-in path is:

https://openwork.example.com/sso/<singleOrgSlug>

For SAML, OpenWork shows the generated ACS URL and metadata URL after the SAML connection is registered. Use those values in the IdP rather than guessing. OpenWork rejects unsigned or weak SAML responses, so configure the IdP to sign assertions.

After SSO is configured, root sign-in shows the SSO-only experience for the single organization. Password sign-in for that organization is rejected.

11. Troubleshooting

Symptom	Likely cause	Fix
Ingress does not reconcile	HTTP load balancing add-on is disabled or Ingress annotation is wrong	Keep HTTP load balancing enabled and use kubernetes.io/ingress.class: gce
Backends are unhealthy	GKE load balancer health checks do not match OpenWork readiness endpoints	Apply the BackendConfig resources and keep the service annotations from the starter values
Ingress events report TimeoutSec should be less than checkIntervalSec	The backend health-check timeout is greater than or equal to its effective interval	Set checkIntervalSec: 15 and timeoutSec: 5 on both BackendConfig resources
Managed certificate is not Active	DNS does not point at the load balancer or provisioning is still running	Point both hosts at the reserved global IP and wait; check kubectl describe managedcertificate
Migration Job fails to connect to MySQL	Private services access, VPC, credentials, IP, or TLS mode are wrong	Test from mysql-client, confirm the private IP, and confirm GKE and Cloud SQL share VPC reachability
Migration Job logs show self-signed certificate in certificate chain	Strict certificate verification is being used without the cloud MySQL CA bundle	Use ?sslaccept=accept for the smoke path or mount/configure the CA bundle before strict verification
ImagePullBackOff from GHCR	Private image or missing pull token	Add imagePullSecrets
Browser auth loops or CORS errors	Public origins do not match DNS/TLS	Set webOrigin, apiOrigin, corsOrigins, betterAuthTrustedOrigins, and authCallbackUrl to the final HTTPS domains
SSO callback rejected	IdP callback URL does not match OpenWork	Use the callback/ACS URL shown by OpenWork for that org/provider
SSO settings show Enterprise gating	DEN_PLAN_GATING_ENABLED=true or org is not entitled	Leave plan gating off for self-host smoke tests, or grant enterprise entitlement

12. Cleanup

For a disposable test:

helm uninstall openwork-ee -n openwork-ee
gcloud compute addresses delete openwork-ee-ip --global
gcloud container clusters delete "$GKE_CLUSTER" --location "$GCP_REGION"
gcloud sql instances delete "$SQL_INSTANCE"

Delete DNS records, retained backups, private service access ranges, and any IAM resources if they were only for the test.