Deployment overview

A connector is a batch job: it runs on a schedule, pulls from a source, pushes to Glean, and exits. glean-idx deploy generates the scaffolding for that — a Dockerfile, an entrypoint, and Terraform for a Kubernetes CronJob on GKE or EKS.

glean-idx deploy init --cloud gcp \
  --connector-class CompanyWikiConnector \
  --connector-factory create_connector

danger

The generated infrastructure has not been validated end to end against a real cloud account. Treat the output as a starting point to review, not as production-ready IaC. Have someone who owns the target account read the Terraform before applying it. Broader validation is tracked in issue #104.

What gets generated

File	Purpose
glean_deployment.yaml	Deployment configuration — you edit this.
Dockerfile	Container image for the connector.
.dockerignore	Build context exclusions.
run.py	Entrypoint that imports and runs your connector.
main.tf, variables.tf	Terraform for the CronJob, service account, and secrets.
.env.example	Template for connector credentials.

Options for init:

Flag	Default	Purpose
--cloud	required	gcp or aws.
--connector-name	current directory name	Used for resource naming.
--connector-class	MyConnector	Class the entrypoint instantiates.
--connector-module	connector	Module to import it from.
--connector-factory	—	Optional zero-argument factory in the connector module.
--output-dir	.	Where to write.

Configuration

glean_deployment.yaml drives generation. The placeholders written by init must be replaced before anything works.

Key	Default	Notes
connector_name	—	Resource naming.
connector_class / connector_module	—	What the entrypoint imports.
cloud	—	gcp or aws.
region	us-central1 / us-east-1	Cloud region.
cluster_name	placeholder	Target Kubernetes cluster.
namespace	default	Kubernetes namespace.
cpu	500m	Pod CPU request/limit.
memory	—	Pod memory request/limit.
cron_schedule	—	Standard cron expression.
indexing_mode	—	full or incremental.
project_id, artifact_registry_repo, service_account_name	—	GCP only.
account_id, ecr_repo, iam_role_name	—	AWS only.

A common pattern is two CronJobs from one image: incremental on a short schedule, full nightly. See Indexing modes.

Workflow

1

Generate — glean-idx deploy init

Writes the Dockerfile, entrypoint, Terraform, and config template.

2

Configure

Edit glean_deployment.yaml (cluster, region, registry, schedule), then cp .env.example .env and fill in credentials.

3

Build and push — glean-idx deploy build --push

Builds the container image and pushes it to your registry.

4

Upload secrets — glean-idx deploy secrets upload

Reads .env, writes each secret to the cloud provider, and atomically records the validated environment keys in local .glean_secret_keys.

5

Apply — glean-idx deploy apply

Reads that manifest and passes its keys to Terraform at apply time so IAM and the CronJob name the exact runtime secrets.

6

Verify

glean-idx deploy status and glean-idx deploy logs --follow, then confirm a document with glean-idx document status --datasource NAME --document TYPE ID. A CronJob that exits zero has not necessarily indexed anything.

Build and push the image before uploading secrets. The local manifest is created by secrets upload, excluded from the Docker build context, and consumed by deploy apply; it is not baked into the image. Uploading before build is neither required nor recommended.

warning

apply prompts once before invoking terraform apply -auto-approve; --yes skips that prompt for unattended use. Run terraform plan in the generated terraform/ directory and review it before applying.

Secrets

Credentials are read from the environment at runtime and injected from the cloud secret store. Never bake them into the image or commit .env.

secrets upload validates environment-variable keys and writes their sorted names to .glean_secret_keys. At apply time, Terraform grants the workload access to each exact declared secret: one secret-level IAM member per key on GCP, or the exact resolved secret ARNs on AWS. The generated runtime receives the declared key list and fetches those values directly. It has no permission to enumerate secrets and does not discover them by connector-name prefix.

glean-idx deploy secrets list remains an operator command; the deployed connector does not use it. Startup fails if any declared secret cannot be loaded. A missing or empty manifest produces a secretless deployment.

Teardown

glean-idx deploy destroy

Requires two confirmations. --keep-secrets preserves stored secrets.

Running elsewhere

Nothing about the SDK requires Kubernetes. A connector is a Python process that needs two environment variables, so it runs anywhere: a cron entry on a VM, a Lambda or Cloud Run job, an Airflow task, a GitHub Actions schedule.

The generated Dockerfile is useful even if you discard the Terraform. What matters is that the job runs on a schedule, has credentials, exits non-zero on failure, and is monitored — see Observability.