Restart apps stranded by an in-place sidecar restart (#149) #17

Merged

backmeupplz merged 2 commits from fix/149-sidecar-restart-rejoin into master

2026-10-20 23:59:08 +00:00

backmeupplz commented

2026-10-20 23:42:53 +00:00

(Migrated from github.com)

Fixes agentboard #149.

Problem

An app with network_mode: service:ts-sidecar (or service:gluetun) runs as container:<owner id>. When the owner restarts in place (same ID, via its unless-stopped policy or compose restart ts-sidecar, which regenerate_all_serve_configs also does), it gets a fresh network namespace. The app stays in the old one, so it still reports healthy while the sidecar's proxy gets connection refused and the app goes dark. This happened to vaultwarden on 2026-10-20.

Fix

src/netns.rs adds a watchdog, spawned next to the scheduler in serve(). Every 30s it:

lists running containers and picks the myground-* ones whose HostConfig.NetworkMode is container:<id> with a running owner;
inspects State.StartedAt for each dependent and owner;
restarts each dependent whose owner started after it, once the owner has been up for at least 20s (so compose can finish its own restarts first).

Owners that are missing or stopped are skipped, because a restart can't rejoin them. Unknown start times are skipped too. After a restart, the dependent's start time is later than the owner's, so it can't loop.

Tests

cargo test: all suites pass (lib 309, api_tests 83, …).
New unit tests in netns::tests cover parsing, the stranded/not-stranded decision, the grace period and unknown times.
New Docker e2e tests/netns_e2e.rs (ignored by default, cargo test --test netns_e2e -- --ignored) runs a busybox owner+app pair. It restarts the owner, checks that the app is unreachable from the owner's namespace, runs the watchdog and checks that the app is reachable again and isn't restarted a second time. It passed locally on Docker Desktop.

Version bumped to 0.1.105.

Fixes agentboard [#149](https://agentboard.icefish-betta.ts.net/#149). ## Problem An app with `network_mode: service:ts-sidecar` (or `service:gluetun`) runs as `container:<owner id>`. When the owner restarts in place (same ID, via its `unless-stopped` policy or `compose restart ts-sidecar`, which `regenerate_all_serve_configs` also does), it gets a fresh network namespace. The app stays in the old one, so it still reports healthy while the sidecar's proxy gets `connection refused` and the app goes dark. This happened to vaultwarden on 2026-10-20. ## Fix `src/netns.rs` adds a watchdog, spawned next to the scheduler in `serve()`. Every 30s it: - lists running containers and picks the `myground-*` ones whose `HostConfig.NetworkMode` is `container:<id>` with a running owner; - inspects `State.StartedAt` for each dependent and owner; - restarts each dependent whose owner started after it, once the owner has been up for at least 20s (so compose can finish its own restarts first). Owners that are missing or stopped are skipped, because a restart can't rejoin them. Unknown start times are skipped too. After a restart, the dependent's start time is later than the owner's, so it can't loop. ## Tests - `cargo test`: all suites pass (lib 309, api_tests 83, …). - New unit tests in `netns::tests` cover parsing, the stranded/not-stranded decision, the grace period and unknown times. - New Docker e2e `tests/netns_e2e.rs` (ignored by default, `cargo test --test netns_e2e -- --ignored`) runs a busybox owner+app pair. It restarts the owner, checks that the app is unreachable from the owner's namespace, runs the watchdog and checks that the app is reachable again and isn't restarted a second time. It passed locally on Docker Desktop. Version bumped to 0.1.105.