Running Colibri

Four services, three of which need to reach each other:

Set the service URLs explicitly. Colibri can derive them from the hostname when it recognises the platform it is running on, and will refuse to guess in production otherwise — a self-hosted deployment must set API_URL, APP_URL and WEBSITE_URL. APP_URL also drives the trusted origins that sign-in and CORS depend on, so a wrong value here surfaces as authentication failing rather than as a misconfiguration.

The app reaches the API twice, over different paths: server-side through COLIBRI_API_INTERNAL_URL (private network), and from the browser through the API's public URL. Both must be set. The API reaches the worker at IMPORTER_API_URL, authenticated with IMPORTER_API_TOKEN.

/health is the liveness endpoint. It answers as soon as the process is serving and touches no dependency, so it is safe for a load balancer to poll — and it deliberately keeps answering in setup mode, so a platform does not kill a deployment before anyone can read the wizard.

/admin is operator surface. Restrict it at the ingress to your private network if you can; it is credential-gated either way, but there is no reason for it to be reachable from the internet.

The API and app are small and mostly I/O-bound — they hold no dataset in memory and scale horizontally without coordination. Start with the smallest instances your platform offers and scale on request latency.

The worker is the one that matters. It processes one materialization at a time — concurrency is capped at one request per instance deliberately, because a job loads a dataset into DuckDB and memory is the binding constraint. So:

• Memory per worker is set by your largest single datasource, not by how many you have. A worker that can process your biggest source can process all of them.
• Number of workers is set by how many imports you need to run at once. Throughput scales by adding instances, never by making one bigger.
• Imports are long-running; the worker is configured for a long idle timeout and shuts down on SIGTERM. Give it a termination grace period long enough to finish or abandon a job cleanly.

The API needs disk for staged datasource files — budget a few GB and let it grow. Postgres holds metadata, not datasets, and stays small relative to the object store, which holds materialized output and is where storage actually accumulates.

Database migrations run outside the application, as a release step before the new version serves traffic. The API does not migrate on boot in production, so a new version must not be started until migrations for it have been applied.

1. Back up Postgres (see below). Migrations are not reversible.
2. Run the migration step for the new version against your database.
3. Deploy the API, then the worker, then the app.
4. Check /admin. It reports whether each dependency still answers, which is the fastest way to catch a credential or URL that did not survive the upgrade.

Migrations are written to be applied to a live database, so a brief overlap between versions is expected and safe. Rolling back the application is safe; rolling back the database is not — restore from a backup rather than attempting to reverse a migration.

Four things hold state, and they are not equally replaceable:

Colibri's database and the vault must be restored together. A connection row points at the secret that holds its credentials; restoring one without the other leaves connections that reference secrets which no longer exist. Take both at the same point in time.

Back up your configuration too, not just data. BETTER_AUTH_SECRET in particular is not derivable — a restore with a different value signs out every user and invalidates outstanding invitations. The vault credentials are equally unrecoverable if they exist nowhere but a deleted task definition.

Rehearse a restore before you need one. The failure people discover during an incident is almost never a missing backup; it is a backup that restores into a deployment configured slightly differently.

Self-hosted deployment is still being shaped. Everything here is accurate for the current release, but it has not been proven against a real customer install — expect it to get more specific as it is.