AI penetration test, tenth run (2026-11-06)

A short targeted step 4 of docs/ai-pentest.md on commit f5e3aab (current main, SPEC v0.37), the fixes for the ninth run (reports/AI pentest, ninth run (2026-11-06).md): area G (off-chain parts) only, scoped to what v0.37 changed in the service, the web app and docker-compose.yml since dc3e837, and whether each of the ninth run's five fixes holds. No Solidity changed (git diff --quiet dc3e837 f5e3aab -- contracts/src), so the contracts were out of scope. One fresh attack agent, with the procedure's prompt word for word and the brief narrowed to the fixes; it was asked to prove every finding on the real local stack (the service's harness: an Anvil fork of Arbitrum Sepolia with real Kleros, a scratch Postgres with the real migrations, the real app and keeper, and the real main.ts as a child process).

The terms of this round (the session's, after the user left the next steps to it on 2026-11-06). Attack only the v0.37 fixes: the sponsored-fee reservations under a lock and the cleanup of a fee reserved but never sent (service/src/sponsor.ts); the keeper's order, its payment of kept answers past the budget in the last third of their fee period, and admins' approvals within the same daily limit (service/src/answers.ts); the relayer's hash at broadcast, the shutdown that finishes the sends in flight, the database pool's error listener, the stop grace period, the rescue ops alert and the reminders for kept answers (relayer.ts, main.ts, db.ts, docker-compose.yml, ops.ts, alerts.ts); and the web app's kept-answer card, fee button, admin page and error texts. If it found nothing above Low, its findings would be fixed with regression tests and the loop closed; a Medium or worse would have been fixed and attacked once more.

Part 1. Summary

Result

No finding, at any severity, and no contract change. All five of the ninth run's fixes hold on the real stack:

• R9-G-2 (Medium): kept answers go before fees set aside for answers still to come, answers to a buyer's challenge first, so unanswered sham disputes no longer take each freed fee ahead of a real answer; the keeper pays a kept answer in the last third of its fee period past the budget, within SPONSOR_RESCUE_DAILY_WEI; the urgent ops alert fires when even that can't pay it; the deal page, the reminders and the fee button tell the party the fee isn't paid yet. To starve a real answer now, an actor must have yescrow pay its own sham answers or make real registry changes, which uses up both daily limits (about $18 a day to yescrow, paid to Kleros) and still leaves the urgent alert, the operator's approval and the party's own payment: the residual SPEC §14.17 and docs/audit.md §5 state.
• R9-G-5 (Medium): a fee reserved but never sent is dropped once the escrow shows the relayer didn't pay that side, never while this process is sending it or after the relayer did pay; the hash is recorded when the transaction goes out; a SIGTERM finishes the send in flight; a Postgres restart leaves the service running.
• R9-G-1 (Low): admins' approvals and the keeper's payments past the budget share one daily limit, checked inside the locked reservation; approvals keep the relayer's reserve and get RESCUE_LIMIT past the limit; the rescue alert names each approving wallet; docs/keys.md and SPEC §12 describe the power and its cap.
• R9-G-4 (Low): a reservation is one transaction under an advisory lock, refused by a rollback that leaves nothing behind, so no request can delete a fee another one claimed.
• R9-G-3 (Info): the G-1 regression runs on a scratch database, and scripts/check.sh runs the light off-chain proofs with the stop-and-restart test.

One observation, not a finding: when a kept answer's party gets ETH in the last third of the fee period, the keeper records "your wallet has enough ETH to pay the fee" as the reason, so the urgent ops alert reads "the keeper can't pay it past the budget: your wallet has enough ETH…". It is accurate (yescrow leaves the fee to a party that can pay it), the party sees its fee button, and nothing is lost. Left as is.

All findings

None.

Three ideas stayed unconfirmed (in the area report): two service processes overlapping during a deploy (Compose stops the old one first; the lock and the chain checks would still prevent a double payment); whether the keeper's transfer and payout sends abandoned by a crash are reconciled after a restart, as answers and appeal stakes are; and an outsider growing the service's memory to an OOM kill (no mem_limit), which a restart now recovers from.

The regression run

The service's tests on f5e3aab before it was committed (in service/, node --test --test-concurrency=1 --test-timeout=900000 test/*.test.ts, with the database from scripts/test-db.sh): 157 tests, 157 pass, 0 fail, among them the four tests for the ninth run's fixes on the fork and the new alert and ops tests; the web app's unit tests, 84 pass; the 16 light off-chain proofs scripts/check.sh now runs, all pass; and scripts/pentest/ninth/G9_fixed_stop_and_restart.ts, PASS. With the fixes undone one at a time, each of the four fork tests failed. The attack agent ran the four fork tests, the web tests, G_sponsor_drain.ts and the stop-and-restart test again on its copy: all pass.

Limits

• Short and targeted, on the v0.37 changes only; nothing else was attacked in this round.
• No new proof was written. With no finding, the attack agent confirmed the fixes with the project's own regression tests and reasoned about variants (listed in "Checked and found sound"); nothing new was run against them beyond those tests.
• No verifier ran: there was no finding to verify.
• Not independent of the author. Launched from the session that wrote the fixes; the attack agent had a fresh context and the procedure's prompt word for word, on Claude Fable 5.1 at maximum reasoning effort. No model from another vendor was run.
• A fork, not mainnet, and the web app read and tested in node, not in a browser or a wallet.
• No findings is not proof of no bugs.

Part 2. Area report

Area G: off-chain parts

AI penetration test, tenth run, area G: off-chain parts (commit f5e3aab, the ninth run's fixes)

The attack agent's report, as it wrote it.

A short, targeted step 4 of docs/ai-pentest.md on commit f5e3aab (current main, SPEC v0.37), area G (off-chain), narrowed to the v0.37 fixes for the ninth run's five findings (R9-G-1…R9-G-5) and whether each holds. Work was done only in the exported copy. I read SPEC.md (§5, §8, §12, §14.17), docs/audit.md (§3, §5, §6f), docs/keys.md, the full CHANGES-since-dc3e837.diff, and the changed sources (service/src/sponsor.ts, answers.ts, relayer.ts, main.ts, db.ts, ops.ts, alerts.ts, config.ts, api/app.ts; web/lib/deal-state.ts, web/lib/errors.ts, web/components/deal/DealPage.tsx, steps.ts, web/components/admin/Admin.tsx; docker-compose.yml), plus the escrow fee-payer/authorize paths they depend on. I then ran the v0.37 regression proofs on the real stack (the service's Anvil-fork harness with real Kleros and a scratch Postgres from scripts/test-db.sh, the real main.ts as a child process, and the web pure-function tests). Contracts are unchanged and out of scope.

Result

No finding. No Critical, High, Medium, Low or Info weakness was confirmed. All five of the ninth run's fixes hold on the real stack, and the one residual that remains (R9-G-2's bounded, paid starvation) matches what SPEC §14.17 and docs/audit.md §5 already state. I did not change contracts/src, service/src or web/, and I wrote no new source. Because there was no new finding to prove, I wrote no new PoC under scripts/pentest/tenth/; instead I re-ran the project's own v0.37 regression proofs as the confirmation evidence below.

Unconfirmed

• Two service processes sharing the relayer key during a deploy overlap. stop_grace_period: 150s lengthens the old container's drain. If an orchestration ever started the new container before the old one finished (Compose's default up/restart is stop-then-start, so this normally does not happen), the per-process in-memory sending set would not coordinate the two, and only the Postgres advisory lock would. I could not construct a double-pay from this: clearUnsent reads live chain state and refuses to drop a row once openerFeePayer/otherFeePayer == relayer, and a second payArbitrationFee on an already-answered deal reverts at simulateContract before any ETH is sent. Not reachable in the documented single-container deploy; not proven.
• Keeper/transfer/payout loop sends abandoned on SIGKILL. The ninth run left open whether the keeper's transfer and payout sends (not the answer send) are reconciled after a crash the same way clearUnsent/appeals.ts reconcile answers and stakes. v0.37 touched only the answer/fee path; I did not test the transfer/payout paths, carried from the ninth run's unconfirmed list.
• OOM with no mem_limit. The service container still has no mem_limit; whether an outsider can grow its memory to an OOM kill (which, like any crash, is now recovered by clearUnsent on restart) was not tested. Same status as the ninth run.

Checked and found sound

Each ninth-run fix, whether it holds, and why:

• R9-G-1 (admin approvals uncapped, beyond docs/keys.md) — fix holds. Approvals and the keeper's own past-budget payments now share one daily limit SPONSOR_RESCUE_DAILY_WEI, enforced inside reserve under the advisory lock (sponsor.ts:209, rescuedToday counts approved_by IS NOT NULL net of refunds). approveAnswer calls rescueRefusal first, which also keeps the relayer's fee + 0.001 ETH reserve, and returns RESCUE_LIMIT (409) past it (answers.ts:250). The rescue ops alert names each approving wallet (ops.ts). docs/keys.md (admin and relayer rows), SPEC §12 (:451), .env.mainnet.example (SPONSOR_RESCUE_DAILY_WEI=4e15) and docs/mainnet.md:110 all now describe this power and its cap; a leaked admin wallet can make the relayer pay at most the rescue limit a day. Proof: the v0.37 api.test.ts subtest "pays a kept answer past the budget … within a limit admins share (R9-G-2, R9-G-1)" asserts the keeper rescue draws the limit, the admin approval is refused with RESCUE_LIMIT when the limit is used up, and goes through after the limit is raised. PASS (command below).
• R9-G-2 (free starvation of a no-ETH party's answer) — fix holds; residual is as documented. Answers.tick now sends kept (signed) answers before placing any set-aside, answers to a buyer's challenge first (answers.ts:167-170), so the attacker's unanswered sham disputes can no longer take each freed fee ahead of the real answer; a kept answer in the last third of its fee period is paid by the keeper past the budget within the rescue limit (rescueDue, answers.ts:183); an answer-due: urgent ops alert is emailed at once when even that can't pay it (answerProblems, answers.ts:280). The deal page card now says the fee isn't paid yet and the deadline counts (deal-state.ts:333-339, tone warn), the day/hour reminders keep coming with honest text (alerts.ts:50-57,145-153, answerKept populated from kept_answers in alerts.ts:296), and the fee button returns once the party's wallet holds the fee (DealPage.tsx:429-432, and steps.feeStep falls through to a wallet-sent direct step when !sponsored && !keepsAnswer, so what the button says matches what is signed). I confirmed a free variant is no longer possible: to starve a real answer an attacker must get yescrow to pay (send) its own sham answers or make real registry changes, which uses up both daily limits (≈ $18/day to yescrow, paid to Kleros, bounded and visible to the rescue/sponsor alerts) and still leaves the urgent alert, manual approval, and the party's own self-pay — exactly SPEC §14.17's "What remains" and docs/audit.md §5. I also verified a kept answer and a set-aside can never coexist (a set-aside makes sponsorship return sponsored, so the answer is sent, not kept), so no path double-reserves. Proof: api.test.ts "sends kept answers before it sets fees aside, answers to a challenge first (R9-G-2)" and the web tests. PASS.
• R9-G-3 (regression proofs not running) — fix holds. G_sponsor_drain.ts was repaired to run on a real scratch Postgres and is now wired into scripts/check.sh along with the other light off-chain proofs (and G9_fixed_stop_and_restart.ts). Proof: node ../scripts/pentest/G_sponsor_drain.ts → exit 0, "Regression for G-1 OK". PASS.
• R9-G-4 (over-budget release deleting a claimed fee) — fix holds. reserve runs the claim/insert/budget-check in one transaction under pg_advisory_xact_lock (sponsor.ts:166-212), returning false to ROLLBACK with no leftover row, so no DELETE can remove a fee another request claimed; the budget is a hard ceiling again. release is now reached only from the single request holding the sending lock, and only when the tx never went out or reverted (ETH returned). Proof: api.test.ts "takes reservations one at a time, so a refused one never takes another request's fee with it (R9-G-4)" forces the exact interleaving on real Postgres. PASS.
• R9-G-5 (a stopped service blocks a party for good) — fix holds. A reserved-but-unsent row is dropped once the chain shows the relayer did not pay that side (clearUnsent, sponsor.ts:237-249, called from sponsorship and sendSponsored); the hash is recorded at broadcast via relayer.call's new onSent (relayer.ts:97); the loops finish their in-flight iteration on SIGTERM (main.ts:33-52,109-119) within stop_grace_period: 150s; and the pool has an 'error' listener so a dropped DB connection no longer crashes the service (db.ts). I verified clearUnsent cannot drop a genuinely paid fee (it keeps any row whose fee payer is the relayer and any row currently being sent) and cannot double-pay in a single process (the sending guard blocks a second send, and once paid the deal is Disputed). Proof: G9_fixed_stop_and_restart.ts drives the real main.ts on the fork — a SIGTERM mid-send finishes and records it; a SIGKILL mid-send leaves the fee unsent and the answer goes through after restart; a Postgres restart leaves the service running. PASS.
• Deployment settings match the SPEC and .env.mainnet.example: SPONSOR_DAILY_WEI and SPONSOR_RESCUE_DAILY_WEI both 0.004 ETH, deal share 5000 bps, min age 6 h, relayer funded 0.008 ETH = budget + rescue. The keeper's auto-rescue and sponsorship both stop below the relayer's fee + 0.001 ETH reserve, so automatic spending cannot drain the relayer below its gas reserve, and the relayer-low alert still fires.
• Minor, non-exploitable observation (no money, not a finding): when a kept answer's party acquires ETH and the deal is in its last third, the keeper sets kept_answers.reason to "your wallet has enough ETH to pay the fee", so the urgent answer-due: ops alert reads oddly ("the keeper can't pay it past the budget: your wallet has enough ETH…"). It is accurate (yescrow declines because the party can pay), the party sees the self-pay button, and nothing is lost.

Commands run (from the copy's root; all printed PASS / exit 0): - eval "$(scripts/test-db.sh start)" then, in service/, systemd-run --user --scope -q -p MemoryMax=2500M -p MemorySwapMax=2G -- node --test --test-concurrency=1 --test-timeout=900000 --test-name-pattern="ninth run" test/api.test.ts → 4/4 pass (R9-G-1/2/4/5). - cd web && node --test lib/deal-state.test.ts lib/errors.test.ts → 20/20 pass (R9-G-2 web texts, errors). - cd service && TEST_DATABASE_URL=… node --no-warnings ../scripts/pentest/G_sponsor_drain.ts → exit 0 (R9-G-3 repair). - TEST_DATABASE_URL=… systemd-run --user --scope -q -p MemoryMax=2500M -p MemorySwapMax=2G -- node --no-warnings scripts/pentest/ninth/G9_fixed_stop_and_restart.ts → exit 0, PASS (R9-G-5 end to end).

Not done

• No new PoC files were written under scripts/pentest/tenth/: no new finding was confirmed, so there was nothing new to prove. The confirmation used the project's own v0.37 regression proofs above.
• One heavy fork proof per run only. I ran the two fork-backed confirmations (the api.test.ts v0.37 block and G9_fixed_stop_and_restart.ts) and the DB-backed G_sponsor_drain.ts; I did not run the full service suite (per the server limits) or next build.
• The two unconfirmed leads above (deploy-overlap double-process; transfer/payout crash reconciliation and OOM) were reasoned, not reproduced.
• The web dialogs were read and their pure logic tested in node, not driven in a browser or wallet.

The test database was started and stopped; no live system was touched, and no transaction was sent to any network (the Anvil fork read only from the public Arbitrum Sepolia RPC).