Bound deletion-audit capacity by live parts and report refusal reasons (#84) #36

Merged

backmeupplz merged 7 commits from fix/84-audit-capacity-undelivered into main

2026-10-12 17:37:41 +00:00

backmeupplz commented

2026-10-12 17:08:30 +00:00

(Migrated from github.com)

Fixes agentboard #84: Jev stopped deleting all spam because the deletion-audit outbox was "full".

Root cause

PostgresDeletionAudit.prepare refused when COUNT(*) FROM deletion_audit_parts + new parts > capacity (10,000). That count included parts that were already finished (sent, failed deletions, unknown/terminal sends), and those are only removed at 7-day retention expiry. At about 5k deletions/day the retained rows reached 10k, and after that every deletion was skipped (fail-closed).

Change

Capacity counts only live parts (LIVE_PARTS_SQL): parts that are pending/sending, under an intent/pending/sending parent that has not expired, i.e. work that may still be sent. Finished rows stay for dedupe and inspection until retention expiry but no longer occupy capacity. Excluding only sent would not be enough: the live DB holds ~1k delete_failed parts. Dedupe/fencing, retention, the sink lock and fail-closed DB errors are unchanged.
Distinct refusal reasons: prepare takes an optional refused(reason) callback with fixed reasons invalid | closing | duplicate | capacity | database. It fires once, after COMMIT settles. The bot logs it as reason on deletion_audit outcome=intent_unavailable_delete_skipped (unavailable when none was reported). health() gains capacityRefused.
Retained storage no longer has a hard row limit, so the hot paths stay off finished rows:
- Migration 006 adds partial indexes deletion_audit_parts_live, deletion_audit_outbox_live and deletion_audit_outbox_leased, creating them only when missing. It takes the sink row first, so it cannot deadlock a live replica.
- Startup skips 004/005 once the parts table exists. Their CREATE INDEX IF NOT EXISTS locked tables on every start, and the 005 backfill scanned every row. Startup still re-inserts the sink row.
- Expiry uses the expires_at index (statement_timestamp()) and deletes at most 100 reports per pass.
Docs updated (docs/audit-outbox.md).

Deploy note

On the first start, 006 creates the 3 partial indexes once (current tables hold ~3.7k parts, so this takes milliseconds). Later starts take no table locks.

Tests (integration, real PostgreSQL)

At default capacity: 10k sent parts plus 2k other finished parts do not block a new intent, and a live backlog reaching 10k still refuses with reason capacity. Delivery frees capacity immediately. This test fails on main.
Partly-sent and in-flight multipart reports count exactly their unsent parts. This test catches each predicate mutation.
With 100k finished rows, startup succeeds while another connection holds ROW EXCLUSIVE on both tables (fails with the old unconditional 004/005). The capacity query plan uses both live indexes with no Seq Scan, and prepare → confirm → send works.
The index upgrade runs beside a replica that keeps admitting intents, and a deleted sink row is restored on restart.
Reasons are asserted for duplicate, database, closing and invalid. A bot test asserts the logged reason.
bun run check: 498 pass, 0 fail (CI green).

Fixes agentboard [#84](https://agentboard.icefish-betta.ts.net/#84): Jev stopped deleting all spam because the deletion-audit outbox was "full". ## Root cause `PostgresDeletionAudit.prepare` refused when `COUNT(*) FROM deletion_audit_parts` + new parts > capacity (10,000). That count included parts that were **already finished** (sent, failed deletions, unknown/terminal sends), and those are only removed at 7-day retention expiry. At about 5k deletions/day the retained rows reached 10k, and after that every deletion was skipped (fail-closed). ## Change - **Capacity counts only live parts** (`LIVE_PARTS_SQL`): parts that are `pending`/`sending`, under an `intent`/`pending`/`sending` parent that has not expired, i.e. work that may still be sent. Finished rows stay for dedupe and inspection until retention expiry but no longer occupy capacity. Excluding only `sent` would not be enough: the live DB holds ~1k `delete_failed` parts. Dedupe/fencing, retention, the sink lock and fail-closed DB errors are unchanged. - **Distinct refusal reasons:** `prepare` takes an optional `refused(reason)` callback with fixed reasons `invalid | closing | duplicate | capacity | database`. It fires once, after COMMIT settles. The bot logs it as `reason` on `deletion_audit outcome=intent_unavailable_delete_skipped` (`unavailable` when none was reported). `health()` gains `capacityRefused`. - **Retained storage no longer has a hard row limit, so the hot paths stay off finished rows:** - Migration 006 adds partial indexes `deletion_audit_parts_live`, `deletion_audit_outbox_live` and `deletion_audit_outbox_leased`, creating them only when missing. It takes the sink row first, so it cannot deadlock a live replica. - Startup skips 004/005 once the parts table exists. Their `CREATE INDEX IF NOT EXISTS` locked tables on every start, and the 005 backfill scanned every row. Startup still re-inserts the sink row. - Expiry uses the `expires_at` index (`statement_timestamp()`) and deletes at most 100 reports per pass. - Docs updated (`docs/audit-outbox.md`). ## Deploy note On the first start, 006 creates the 3 partial indexes once (current tables hold ~3.7k parts, so this takes milliseconds). Later starts take no table locks. ## Tests (integration, real PostgreSQL) - At **default** capacity: 10k sent parts plus 2k other finished parts do not block a new intent, and a live backlog reaching 10k still refuses with reason `capacity`. Delivery frees capacity immediately. This test fails on `main`. - Partly-sent and in-flight multipart reports count exactly their unsent parts. This test catches each predicate mutation. - With 100k finished rows, startup succeeds while another connection holds ROW EXCLUSIVE on both tables (fails with the old unconditional 004/005). The capacity query plan uses both live indexes with no Seq Scan, and prepare → confirm → send works. - The index upgrade runs beside a replica that keeps admitting intents, and a deleted sink row is restored on restart. - Reasons are asserted for `duplicate`, `database`, `closing` and `invalid`. A bot test asserts the logged `reason`. - `bun run check`: 498 pass, 0 fail (CI green).