Skip to content

How to run several worktrees (or several agents) on one machine#

By default every dev environment on a machine shares the same Docker names: the product-opener network, the po_users / po_orgs / po_export_files / po_off_product_images volumes, the openfoodfacts-server/*:dev images, host port 80, and the MongoDB / Redis / PostgreSQL / Keycloak stack from deps/.

That is fine for a single checkout, but it breaks as soon as two worktrees (for example two AI agents editing the same repository) run make dev at the same time:

  • the second one cannot bind host port 80;
  • both join the same network, and Docker DNS answers backend, frontend, postgres, world.openfoodfacts.localhost… for both of them, so requests are round-robined between worktrees;
  • they share the users, orgs and product images volumes, so make hdown in one destroys the other's data, and a change to the .sto format in one breaks the other.

This guide shows how to give each worktree its own names, domain and port.

Quick start#

Once per worktree, from the repository root:

make agent
make dev

That is the whole procedure. You do not have to choose an id and you do not have to check what is free — which is the point, because agents cannot ask each other. make agent:

  • derives an id from the worktree directory name (../off/11790 → w11790),
  • claims a slot for it atomically, so two agents starting at the same moment get different ids rather than both settling on the same one,
  • derives every host port it needs from that slot, and skips a slot whose ports something else already holds,
  • reserves the actual frontend port atomically (including a PORT= override), so two stopped worktrees cannot both register the same port,
  • reuses the same id, slot and frontend port if the worktree already has them, so the URL is stable across re-runs — including while its containers are up, and including an earlier PORT= override.

It prints what it settled on:

🥫 Worktree 'w11790' is now isolated (slot 1).

  .envrc updated (git-ignored, it is a local file).

  URL:        http://world.w11790.openfoodfacts.localhost:8081/
  frontend:   8081
  mongodb:    8181
  postgres:   8281
  redis:      8381
  keycloak:   8481
  smtp4dev:   8681

  dependencies: MongoDB, Redis, PostgreSQL and Keycloak get their own containers
  and their own network (off_shared_network_w11790). Set PO_SHARED_DATA=1 in .envrc
  to go back to one shared stack.

In a second worktree, run exactly the same two commands; the directory name is different, so it gets its own id and port without any coordination.

Seeing what is already taken#

make list-agents

   ID            SLOT  WEB   MONGO  KC   DIRECTORY

 * w11790        1     8081  8181   8481 /home/me/off/11790
   w11790-2      2     8082  8182   8482 /home/me/off/11790-fix

* marks an id whose Docker network still exists, so it is really in use. The registry lives in ~/.cache/off-agents and is per user, so a worktree belonging to another user of the same machine would not be listed — Docker itself remains the authority.

When you do want to choose#

make agent ID=b2              # refused if another worktree already owns 'b2'
make agent ID=b2 PORT=9000    # and pick the port too
make agent PORT=9000          # keep the derived id, choose the port

ID accepts lowercase letters, digits and dashes. Asking for an id that another worktree already owns is refused rather than silently shared, because that would make both worktrees use the same volumes. If the other worktree no longer exists and you want its slot back:

make release-agent ID=b2

To undo isolation for the current worktree, make release-agent removes the generated block and frees the port. The suffixed images, volumes and containers stay on disk until make prune.

Slots are handed out from 1 to 19, and each one reserves seven ports (see What gets isolated), so a 20th concurrent worktree needs a free port range of its own.

What gets written#

make agent writes a generated, git-ignored block to .envrc. It replaces that block on every run and leaves any other line you put in .envrc (such as USER_UID or CPANMOPTS) untouched. Besides PO_AGENT_ID and the ports, the block carries the fully resolved COMPOSE_PROJECT_NAME, PRODUCT_OPENER_DOMAIN, PRODUCT_OPENER_NETWORK and MINION_QUEUE, so host-side processes that never go through the Makefile (notably the Dev Containers host-side Compose invocation, via direnv) see the same settings as make print-agent-config. The Makefile leaves already-resolved values alone, so sourcing .envrc never applies the prefix twice.

It also writes a generated block into each dependency checkout, in deps/openfoodfacts-shared-services/.envrc and deps/openfoodfacts-auth/.envrc. Both of those Makefiles load .env first and then .envrc, and both git-ignore .envrc, which is exactly the override they need. On a fresh worktree without checkouts yet, those files are skipped rather than created early (creating the directories would make clone_deps mistake them for checkouts and skip the clone); make run_deps writes them right after cloning, so a worktree whose deps/ was cloned after make agent is still isolated.

To check what a worktree resolved to:

make print-agent-config

What gets isolated#

Taking w11790 in slot 1 as the example, in addition to everything COMPOSE_PROJECT_NAME already covers:

Resource Default With id w11790
compose project po_off po_off_w11790
default network product-opener product-opener_w11790
minion_db network minion_db w11790_minion_db
users volume po_users w11790_po_users
orgs volume po_orgs w11790_po_orgs
export_files volume po_export_files w11790_po_export_files
product_images volume po_off_product_images w11790_po_off_product_images
images openfoodfacts-server/backend:dev openfoodfacts-server/backend:dev_w11790
domain openfoodfacts.localhost w11790.openfoodfacts.localhost
frontend host port 80 8081
domain of the shared stack openfoodfacts.localhost w11790.openfoodfacts.localhost

The project-scoped volumes (podata, build_cache, node_modules, icons_dist, js_dist, css_dist, pgdata, html_data, products) already followed COMPOSE_PROJECT_NAME, so they are covered too.

The dependencies are isolated as well#

deps/openfoodfacts-shared-services (MongoDB, Redis, PostgreSQL) and deps/openfoodfacts-auth (Keycloak, SMTP4Dev) get their own containers, their own volumes, their own host ports and their own network:

Resource Default With id w11790 (slot 1)
their compose project off_shared / openfoodfacts-auth off_shared_w11790 / openfoodfacts-auth_w11790
shared network off_shared_network off_shared_network_w11790
mongodb port 27017 8181
postgres port 5432 8281
redis port 6379 8381
keycloak port 5600 8481
keycloak management port 5602 8581
smtp4dev port 5605 8681

This is not optional tidiness. Keycloak publishes user-deleted, user-registered and user-updated events to Redis, and every Product Opener instance consumes all three streams (lib/ProductOpener/Redis.pm). With a shared Redis, deleting a test user in one worktree would enqueue delete_user in another, and delete_user_task calls find_and_replace_user_id_in_products, which rewrites product edits in MongoDB to an anonymous user — irreversible, in somebody else's data.

Because the stream names are global, a per-worktree Keycloak alone would not be enough: Redis has to be per worktree too.

MONGODB_HOST, REDIS_URL and KC_SPI_EVENTS_LISTENER_REDIS_EVENT_LISTENER_REDIS_URL need no change: on the per-worktree network, mongodb and redis already resolve to that worktree's containers.

COMMON_NET_NAME, on the other hand, has to be set on both sides. docker/run.yml attaches the Product Opener containers to ${COMMON_NET_NAME} as an external network, and .env defaults it to the shared off_shared_network. So make agent writes COMMON_NET_NAME=off_shared_network_<id> into the worktree's own .envrc as well as into both dependency .envrc files. Without it the containers look for mongodb on the shared network, where this worktree's MongoDB does not exist, and make import_sample_data fails with Temporary failure in name resolution.

MongoDB's WiredTiger cache is lowered to 1 GB per worktree (PO_AGENT_MONGO_CACHE_SIZE overrides it), because the shared default of 8 GB would exhaust the machine with several instances.

Keycloak's advertised URL follows its published port (KC_HOSTNAME=http://auth.openfoodfacts.localhost:<keycloak-port> in the auth .envrc): leaving the shared :5600 default would make Auth.pm redirect logins to another worktree's Keycloak.

What is deliberately not isolated#

  • Nothing that belongs to Product Opener itself. Everything above is isolated, including the dependencies.
  • The registry is per user. It lives in ~/.cache/off-agents, so a worktree owned by a different user of the same machine is not listed. Docker remains the authority, which is what the * marker in make list-agents checks.

Sharing the dependencies again: PO_SHARED_DATA=1#

To go back to one shared MongoDB / Redis / PostgreSQL / Keycloak, put this in .envrc:

export PO_SHARED_DATA=1

It removes the generated dependency blocks, so make run_deps reuses the shared off_shared containers. That is cheaper, and it is the only way to see the production data dump in every worktree — at the price of the cross-talk described above. It shares Keycloak too, because both have to agree on the network name.

make agent PO_SHARED_DATA=1 remembers the setting in .envrc, so a later make dev without the environment variable keeps behaving the same way. To go back to isolated dependencies, set that line to 0 in .envrc (or delete it) and re-run make agent:

sed -i 's/^export PO_SHARED_DATA=1$/export PO_SHARED_DATA=0/' .envrc
make agent

PO_SHARED_DATA=0 make agent does not work, and that is the precedence rule below rather than a bug: the Makefile includes .envrc, and a makefile assignment always wins over an inherited environment variable.

Do not point several worktrees at one DEPS_DIR. The generated dependency .envrc files live inside deps/, so a shared DEPS_DIR cannot be isolated: make agent says so and leaves the dependencies shared. The trade-off is about disk (about 2 GB of dependency repositories re-cloned per worktree) rather than correctness.

/etc/hosts is not needed#

w11790.openfoodfacts.localhost and all its subdomains resolve to 127.0.0.1 on their own in current browsers, because *.localhost is special-cased (RFC 6761). Tools that do not special-case it (curl, wget, some resolvers) still need an entry:

127.0.0.1 world.w11790.openfoodfacts.localhost

make edit_etc_hosts derives the entries from PRODUCT_OPENER_DOMAIN, so it already points at the right worktree.

Only add the entries you actually need; make edit_etc_hosts handles the default domain.

Three guards, so mistakes are loud#

  • If PO_AGENT_ID is set but the host port is still 80, make up / make dev refuse to start instead of silently fighting over the port. Set ALLOW_SHARED_PORT=1 to bypass deliberately.
  • If PRODUCT_OPENER_HOST_PORT and PRODUCT_OPENER_PORT disagree, startup is refused too: Config2_docker.pm derives its server domain from PRODUCT_OPENER_PORT, so a mismatch makes the application generate URLs and redirects pointing at the wrong port. make agent always writes both.
  • If PO_AGENT_ID is set, PO_SHARED_DATA is not, and the dependencies' .envrc does not mention this id, startup is refused: the dependencies would silently stay shared and one worktree would act on another worktree's Keycloak events. make run_deps regenerates those files, so the usual fix is simply to re-run make agent.

How it works#

PO_AGENT_ID is the single knob. make agent writes it to .envrc, and:

  • the Makefile derives COMPOSE_PROJECT_NAME, PRODUCT_OPENER_DOMAIN, MINION_QUEUE and PRODUCT_OPENER_NETWORK by appending the id, but leaves already-resolved values alone (which is what .envrc now carries for direnv-loaded shells and the Dev Containers host-side Compose invocation). It also derives COMMON_NET_NAME up front, so adding PO_SHARED_DATA=1 and running make up does not keep using the isolated network that was loaded from .envrc before run_deps removed it on disk.
  • scripts/dev-agent-env.sh keeps a small registry (~/.cache/off-agents) mapping id to slot and directory. Two different things have to be unique and are enforced differently: the slot by a .slot-<n> lock file that is created once and never rewritten, the frontend port (including PORT= overrides) by a .port-<n> lock file, and the id by writing its registry entry with set -o noclobber, once, at the very end. All three are atomic on POSIX filesystems. If a run fails after claiming, both the id entry (first) and the locks are given back, so the registry never keeps a worktree that has no slot, and a retry never reuses a stale entry without holding a lock.
  • the compose files derive network, volume and image names from PO_AGENT_PREFIX / PO_AGENT_SUFFIX, which the Makefile computes from the id.
  • the compose files derive network, volume and image names from PO_AGENT_PREFIX / PO_AGENT_SUFFIX, which the Makefile computes from the id. Compose's ${VAR:+word} treats an empty but set variable as true, which would produce names such as product-opener_, so the Makefile always passes a real value and the compose files use the empty-safe ${VAR:-} form.

Everything is optional: with PO_AGENT_ID unset, all of these resolve to the historical names, byte for byte.

.github/scripts/check_worktree_isolation.sh asserts all of it in CI: that the default configuration is unchanged, that an isolated worktree is fully suffixed (dependencies and KC_HOSTNAME included), that five worktrees racing for the same directory name still get distinct ids, slots and ports, that a fresh worktree without checkouts gains no dependency .envrc until after the clone, that a re-run without --port keeps its recorded frontend port, that a frontend port reserved by one worktree is refused for another, that a failed claim rolls back both the id entry and the slot, that PO_SHARED_DATA=1 shares again and is remembered, that an id owned by another worktree is refused, and that a DEPS_DIR outside the worktree is reported as shared rather than silently overwritten. You can run it locally at any time.

Shell variables do not override .env for make#

The quick start guide notes that shell variables take precedence over .env. That is true for docker compose, but not for make: the Makefile includes .env, and an assignment in a makefile always wins over an inherited environment variable. So put per-worktree overrides in .envrc (which is included after .env), not in your shell.

If you use direnv, .envrc is loaded automatically, which also makes raw docker compose commands see the prefix.

Other things worth knowing#

  • One branch per worktree. pull_request.yml uses concurrency: {workflow}-{ref} with cancel-in-progress: true, so two agents pushing to the same branch cancel each other's CI run. Separate worktrees normally mean separate branches, which do not collide.
  • DEPS_DIR must stay per worktree. It defaults to $(PWD)/deps, so each worktree re-clones about 2 GB of dependency repositories, and that is the price of isolating the dependencies. Sharing one copy across worktrees leaves MongoDB, Redis, PostgreSQL and Keycloak shared, and make agent warns when you do it. The Makefile does honour an existing DEPS_DIR, so a per-worktree path outside the checkout (for example ~/off-deps-11790) works fine.
  • RAM. Budget roughly one backend container (about 6 GB) plus about 1 GB for the isolated dependencies per worktree. Two or three concurrent worktrees is usually the practical limit; measure before adding more. PO_SHARED_DATA=1 brings the dependency part back down to about 250 MB.
  • Devcontainer. .devcontainer/devcontainer.json forwards only PO_AGENT_ID and the two affixes into the container (app-level settings such as PRODUCT_OPENER_DOMAIN come from Compose itself — forwarding them via ${localEnv:...} would set them to empty when the host hasn't exported them and break URL generation). The host-side compose invocation is what decides container and volume names, so load the worktree's .envrc in your host shell — with direnv, or with set -a; source .envrc / eval "$(make print-agent-config | sed 's/^/export /')" — before reopening the folder in the devcontainer, otherwise the devcontainer silently falls back to the shared names. make agent writes the resolved COMPOSE_PROJECT_NAME, PRODUCT_OPENER_DOMAIN, PRODUCT_OPENER_NETWORK and MINION_QUEUE to .envrc exactly so direnv can provide them.
  • Keycloak sessions are not isolated. Every worktree serves Keycloak on the same hostname (auth.openfoodfacts.localhost, only the port differs) with the same openfoodfacts realm, and browser cookies are scoped by hostname, not port. Logging in on a second worktree therefore overwrites the first one's Keycloak session cookies. Avoid being logged in on two worktrees in the same browser profile at once.
  • Tests keep using their own project name and PO_COMMON_PREFIX=test_. If PO_AGENT_ID is set in your shell they simply become suffixed too, which is harmless: the test project name becomes po_off_w11790_test. The application URLs inside tests stay canonical: the test Compose targets pin PRODUCT_OPENER_DOMAIN, PRODUCT_OPENER_PORT and MINION_QUEUE back to the shared defaults, so expectations such as the callback URL in tests/integration/oidc_signin.t keep matching.

See also#