Fix worker queue fairness during backlog recovery #198

Merged

backmeupplz merged 1 commit from codex/VOI-71-worker-queue-fairness into main

2026-05-21 02:29:43 +00:00

backmeupplz commented

2026-05-21 01:57:10 +00:00

(Migrated from github.com)

Summary

Bound the Windows worker ready queue so downloads cannot build an unbounded stale in-memory backlog ahead of GPU transcription.
Alternate oldest/newest download claim buckets and recheck downloads between transcriptions after queue exhaustion so fresh voice jobs can run between older backlog jobs.
Recover same-worker ready and stale transcribing/processing jobs via /jobs/claim-ready, with documented queue/recovery knobs and log context for scheduling buckets.
Add worker-client and worker-API proofs for no-starvation fairness and stale active-job recovery.

Validation

yarn build-ts
yarn test:worker-client
MONGO=mongodb://127.0.0.1:27018/voicy_voi71_worker_api yarn test:worker-api using temporary mongo:6 Docker container; container stopped afterward
yarn lint

Manual QA

Required after deploy: while production backlog remains non-empty, send a fresh short Telegram voice and confirm the Windows worker claims/processes it promptly. Capture worker log lines showing claimBucket/schedulingBucket; expected behavior is fresh/newest work appearing between older backlog jobs rather than waiting for the ready backlog to fully drain.

## Summary - Bound the Windows worker ready queue so downloads cannot build an unbounded stale in-memory backlog ahead of GPU transcription. - Alternate oldest/newest download claim buckets and recheck downloads between transcriptions after queue exhaustion so fresh voice jobs can run between older backlog jobs. - Recover same-worker ready and stale transcribing/processing jobs via `/jobs/claim-ready`, with documented queue/recovery knobs and log context for scheduling buckets. - Add worker-client and worker-API proofs for no-starvation fairness and stale active-job recovery. ## Validation - `yarn build-ts` - `yarn test:worker-client` - `MONGO=mongodb://127.0.0.1:27018/voicy_voi71_worker_api yarn test:worker-api` using temporary `mongo:6` Docker container; container stopped afterward - `yarn lint` ## Manual QA Required after deploy: while production backlog remains non-empty, send a fresh short Telegram voice and confirm the Windows worker claims/processes it promptly. Capture worker log lines showing `claimBucket`/`schedulingBucket`; expected behavior is fresh/newest work appearing between older backlog jobs rather than waiting for the ready backlog to fully drain.

backmeupplz commented

2026-05-28 21:55:19 +00:00

(Migrated from github.com)

Rework validation for VOI-71 after the task was reopened:

PR #198 is already merged and its CI/proof (test:worker-client) passes, but the merged scheduler does not cover the reopened production failure mode. The new production evidence is a fresh job that had already reached ready and then waited behind a large ready backlog.

The gap is in the merged scheduler:

processAvailableJobs() only alternates oldest/newest for download claims after downloadQueueExhausted (src/workerClient/runWindowsWorker.ts around the download scheduler).
Recovered/ready work is still filled via claimReadyJob(api, 'oldest') only and placed in the special recovered queue (src/workerClient/runWindowsWorker.ts around fillRecoveredReadyJobs).
The API can sort ready claims by newest (claimReadyJobFromBucket supports bucket: newest and sorts by downloadedAt/createdAt), but the worker never uses that for ready backlog recovery.

So this PR should not close VOI-71. The follow-up fix needs explicit fair scheduling for already-ready jobs as well, e.g. alternate oldest/newest claim-ready buckets while backlog exists, and add a regression proof where two old ready jobs exist, a newer short ready job exists, and the newer job is selected before the old ready backlog drains. Live QA still needs a fresh Telegram voice while ready backlog remains non-empty.

Rework validation for VOI-71 after the task was reopened: PR #198 is already merged and its CI/proof (test:worker-client) passes, but the merged scheduler does not cover the reopened production failure mode. The new production evidence is a fresh job that had already reached ready and then waited behind a large ready backlog. The gap is in the merged scheduler: - processAvailableJobs() only alternates oldest/newest for download claims after downloadQueueExhausted (src/workerClient/runWindowsWorker.ts around the download scheduler). - Recovered/ready work is still filled via claimReadyJob(api, 'oldest') only and placed in the special recovered queue (src/workerClient/runWindowsWorker.ts around fillRecoveredReadyJobs). - The API can sort ready claims by newest (claimReadyJobFromBucket supports bucket: newest and sorts by downloadedAt/createdAt), but the worker never uses that for ready backlog recovery. So this PR should not close VOI-71. The follow-up fix needs explicit fair scheduling for already-ready jobs as well, e.g. alternate oldest/newest claim-ready buckets while backlog exists, and add a regression proof where two old ready jobs exist, a newer short ready job exists, and the newer job is selected before the old ready backlog drains. Live QA still needs a fresh Telegram voice while ready backlog remains non-empty.