Two causes of the paired room "keeps stopping" symptom:
- Reviewer approvals worded as "PROCEED" were parsed as 'continue'
(a change request), causing an owner TASK_DONE <-> reviewer PROCEED
ping-pong until the deadlock cap. parseReviewerVerdict() now treats a
leading PROCEED as approval so the turn finalizes after one round.
- When the Codex reviewer was unavailable, the owner's answer was held
for review and the user saw nothing. Now the held owner answer is
emitted with a "review skipped" notice on reviewer_codex_unavailable.
Verified: tsc --noEmit clean; 27 related vitest tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Claude usage poller retried /api/oauth/usage every 60s, but the endpoint
returns 429 with a longer Retry-After window (~93s). Retrying mid-cooldown
re-tripped the limit so the 429s never cleared and usage data never populated.
Record a cooldownUntil from the 429 Retry-After header (5min fallback when
absent) and skip the API until it closes; cleared on success. Dashboard now
shows a "429" indicator instead of a stale value when rate-limited.
Adds regression tests proving the cooldown outlasts the 60s throttle and
releases once the window passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
assign_room (MCP tool + host-side assignRoomInDatabase) and setup
register previously fell back to 'single' when no room_mode was given,
so newly added rooms silently lost the reviewer. Default to 'tribunal'
across the MCP zod schema, the IPC arg fallback, and the DB helper, and
update the ipc-auth expectation accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resolves the gitea/main <-> deployed-line merge:
- DB migrations: renumber gitea's colliding 019/020 to 021/022
(reviewer_failure_count -> v21, turn_progress_text_compat -> v22) so all
four migrations have distinct versions; update ordered list + bootstrap test.
- paired_tasks INSERT: add the missing VALUES placeholder so both new columns
(reviewer_failure_count + arbiter_intervention_count) bind (26 cols/values).
- index.ts: drop duplicate startUsagePrimer import from the auto-merge.
- discord output: keep the deployed pipeline (attachment rejection notice) and
call sanitizeForOutbound at the channel boundary so prose escaping done in
prepareDiscordOutbound is not double-applied; keep gitea's reviewer
silent-failure cap + router markdown-escape helpers.
- usage-primer/codex-warmup: keep the dawn-hold removal over gitea's primer.
Full test suite: only pre-existing env/bun-path failures remain; no merge regressions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reviewer/arbiter/owner Codex turns are no longer deferred during the
[03:00-08:00 KST] dawn window — they dispatch immediately at all hours.
Deletes codex-primer-alignment.ts and every hold call site (gating owner
hold + alignment notice, reviewer/arbiter queue hold, warm-up skip, primer
anchor-lock). The 08:00 primer still fires; it just no longer holds other
consumers. Tradeoff: the 13:00 KST 5h-reset anchoring is no longer enforced.
Also adds a user-visible next-step indicator after owner turns that do not
end the task (review_ready -> reviewer requested, arbiter_requested ->
arbiter called) so a same-looking status line is no longer ambiguous; no
extra line on completion to avoid duplicating the owner's final message.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Scheduled tasks only suppressed phase 'progress', so intermediate
preambles (e.g. "I'll run the watchdog checks.") leaked to the chat
each run while the actual <internal>-wrapped result was correctly
stripped. The interactive path treats intermediate/tool-activity as
silent (toVisiblePhase); align the scheduler to forward only the final
message, with error outputs still falling through to rotation/error
handling.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Show NVIDIA GPU utilization and VRAM used/total in the 서버 status block,
matching the existing CPU/Memory/Disk bar format. Gracefully omitted when
nvidia-smi is unavailable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The staging logic in 9c46cf6 copied any agent-declared file from outside the
room's allowed directories into a safe folder and attached it, which bypassed
the attachment directory allowlist (cross-room isolation / sensitive-file
protection). Removing it.
Kept: appendRejectionNotice / describeRejectedAttachments so rejected
attachments are surfaced in the visible message instead of being silently
dropped. This changes no security behavior — it only adds text when an
attachment was already going to be rejected.
Verified: outbound-attachments + final-delivery + discord tests 70/70, tsc clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Agent-generated files written to an arbitrary working path (e.g. TTS audio
under /home/claude/jarvis-tts) were rejected by validateOutboundAttachments as
"outside-allowed-dirs". The rejection was only logged; the MEDIA: directive had
already been stripped from the text, so the user got a message claiming a file
was attached with no file and no error.
- Stage attachments outside the room's allowed dirs into a safe per-group dir
(data/attachments/outbound/<group>) at the universal delivery choke point, so
the path delivery uses and revalidates is one the validator accepts. Files
already inside an allowed dir are untouched, preserving isolation checks.
- Surface any still-rejected attachment in the visible Discord body via
appendRejectionNotice, so delivery can never again silently drop a file.
Verified: outbound-attachments 22/22, discord 46/46 (incl. new integration test
asserting the notice lands in the sent body), final-delivery 5/5, tsc clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Address reviewer findings on the narrow dawn-hold approach:
- 08:00 release race (the daily failure): the dawn window's clock boundary
lapses at exactly 08:00, so a held reviewer turn released at 08:00:00 could
beat the primer to open the fresh window. Add an in-process anchor lock — the
08:00 primer sets a flag for the duration of its own Codex call, and a short
post-slot grace bridges the gap between the boundary lapse and the flag set.
Held consumers stay held until the primer's anchoring request completes.
- Owner gate now keys on the EFFECTIVE owner backend (configured codex owner OR
a global-failover override to Codex) via getEffectiveChannelLease, not the
static group.agentType — covers failover-to-Codex rooms.
- Dedupe the owner wait-notice to at most once per room per dawn window
(ALIGNMENT_NOTICE_KEY) instead of on every re-poll. Held turns still resume
exactly once: the gate returns without advancing the message cursor.
Note on coverage: the deep common boundary (runAgentForGroup) returns only
'success' | 'error' with no defer state — holding there would consume/lose the
message or trigger handoff/failover, so holds stay at the cursor-safe shallow
gates. Paired reviewer/arbiter turns (incl. scheduled review-ready
reconciliation) and codex-owner turns are covered; generic codex scheduled
tasks (runTask) and Claude->Codex handoffs at dawn remain rare residual edges.
Tests: anchor-lock + grace + slot/notice-key (codex-primer-alignment.test),
failover-owner hold + notice dedupe (message-runtime-gating.test). 27 focused
tests pass; the 9 pre-existing service-routing/migration failures predate this
work (verified at 728a8c2).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Codex 5h usage limit is a FIXED window (verified live: usedPercent climbs
while resetsAt stays put), so it can be anchored — but only if the 08:00 KST
primer is the first Codex request of a fresh window. Codex is otherwise used
near-continuously (reviewers/arbiters every paired room, codex-owner rooms,
warm-up), so 08:00 lands mid-window and the reset drifts (observed 15:47).
Restore the alignment hold from backup/primer-hold-work, but narrowed to a
single dawn window (03:00-08:00 KST) per user direction — the broad all-slots
version was previously reverted. One Codex window is at most 5h, so anything
opened before 03:00 has expired by 08:00, leaving the primer to anchor cleanly
=> 13:00 reset. Other slots (13/18/23) and all daytime hours are untouched.
- codex-primer-alignment.ts: narrow shouldHoldCodexForPrimerAlignment (03-08 KST)
- codex-warmup.ts: skip non-primer (non-forceAttempt) warm-ups during the hold
- message-runtime-queue.ts: defer Codex reviewer/arbiter turns during the hold
- message-runtime-gating.ts: defer fresh Codex-owner room turns during the hold
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Codex primer awaited refreshAllCodexAccountUsage + refreshActiveCodexUsage
before firing, purely to feed the old usage-based warm-up gating. With
forceAttempt the primer fires regardless of usage, so those pre-call refreshes
only added variable latency (the few-hundred-ms jitter that pushed the call off
the exact slot) and a needless dependency on the usage API. Fire first at the
slot; refresh once afterwards for observability (never delaying the call). The
hourly usage collector keeps cached usage current.
Verified: typecheck, build, bundle-smoke; usage-primer + codex-warmup suites pass
(primer now fires before any refresh; refreshAll no longer called).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The scheduled primer reused the opportunistic warm-up gates, so the 18:00 slot
(exactly 5h after 13:00, but firing a few hundred ms later) fell just under
minIntervalMs and was silently skipped as no_eligible_accounts — the primer did
not actually fire at every slot. Add a forceAttempt path: the fixed-slot primer
bypasses stagger, post-failure cooldown, min-interval, and per-account
usage/rate-limit filters, attempting a real Codex command on the active account
at every 08/13/18/23 slot and recording success/failure.
Reviewer/arbiter holding stays removed (per user). This does not pin the reset
clock — a trailing window still slides with later usage — but it guarantees the
"simple task at every fixed slot" the user asked for.
Verified: typecheck, build, bundle-smoke; new regression tests (13:00 success
then sub-5h 18:00 still fires; control without forceAttempt is skipped;
forceAttempt bypasses cooldown/stagger) + usage-primer/codex-warmup suites pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The user asked to NOT defer reviewer/arbiter Codex calls, but instead just fire
a simple primer task at each fixed slot independent of the reviewer. That is
already the live behavior (efc8c00: primer fires at 08/13/18/23 KST + the Claude
binary-path fix). Revert the hold/gating work (e324a1b..ef2b26e) so it can't be
deployed; it is preserved on branch backup/primer-hold-work in case the 08:00
overnight-exhaustion gap later warrants a minimal quiet window.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Address review gaps in the primer-alignment hold so the fixed-slot primer
reliably wins the first-request race for the shared Codex window:
- Extract the time-based hold into a dependency-free leaf module
(codex-primer-alignment.ts) so every Codex path can share it without import
cycles.
- Add a pre-slot hold window (2 min) in addition to the post-slot (5 min) and
the 04-08 KST dawn gap, closing the "request a beat before the slot" gap.
- Hold the dashboard's periodic Codex warm-up too: runCodexWarmupCycle now
skips during the hold unless called with isPrimer (the scheduled primer sets
it), so the warm-up can't anchor the window ahead of the primer.
- Remove the urgent @-mention bypass on the reviewer/arbiter and owner holds —
the reset is anchored unconditionally as requested.
- Reviewer/arbiter hold resumes exactly once after the window via the existing
per-revision claim (poll re-detect; no double run).
Note: the unified lease boundary (syncHostCodexSessionFiles) is synchronous and
throws a terminal "Codex unavailable", so it can't host a clean defer+resume;
gating stays at the async turn-scheduling layer that can re-queue. Honest caveat
unchanged: making the primer first is necessary but not sufficient to pin a
trailing reset.
Verified: typecheck, build, bundle-smoke; full suite 1482 passing with only the
9 pre-existing env-config baseline failures (service-routing/paired-context/
migrate-room owner=claude vs codex-main); new alignment + warmup-hold tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The usage primer fires at 08/13/18/23 KST but could not pin the Codex 5h
reset because reviewer/arbiter turns on the same shared account ran off-slot
and anchored the window first. Add a time-based hold
(shouldHoldCodexForPrimerAlignment) that keeps non-primer Codex turns quiet
during the 04-08 KST dawn gap and for a few minutes after each slot, so the
scheduled primer wins the first-request race. Applied at the reviewer/arbiter
dispatch site (the actual off-slot consumer here) and for codex-owner rooms;
urgent @-mention turns bypass it. Time-based by design — the usage/reset
figures are exactly the unreliable data, so a blunt clock rule is predictable
and testable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live evidence shows the Codex 5h limit is a trailing rolling window whose
reported reset slides forward with continued usage (observed 18:00 → 18:33
over ~33 min). A timed primer therefore cannot pin the reset to a fixed clock
time, and the previous "retry at reported reset" logic was both based on a
wrong fixed-window model and buggy under sliding resets (stale reset target,
single-shot). Revert it.
Keep the verified Claude primer binary-path fix (resolve repo-root
node_modules) and document the real rolling-window behavior so the primer is
correctly framed as a best-effort warm-up + success/failure record.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two evidence-based fixes for the usage-window primer that aligns the 5h
reset to fixed KST slots (08/13/18/23):
1. Claude primer always failed with binary_missing: it only looked under
runners/agent-runner/node_modules, but the SDK platform binary is hoisted
to the repo-root node_modules. Resolve both locations.
2. When a Codex slot lands while the 5h window is still active/exhausted, the
warm-up call fails with a rate-limit and the window only re-anchors at its
natural reset — leaving Codex un-primed until the next 5h slot. Parse the
account's known reset time and schedule a one-shot retry right after it so
the next window anchors close to the slot.
Also corrected a stale comment claiming a 04:00–08:00 gap is enforced by
message-runtime-gating (it is not).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migration 020 (arbiter_intervention_count) was added without updating the
hardcoded expected list in bootstrap.test.ts, breaking its schema-migration
assertions. Add the entry so the canonical `bun run test` suite passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The owner↔reviewer↔arbiter loop could repeat forever: when the arbiter
ruled PROCEED/REVISE/RESET it reset round_trip_count to 0, and nothing
tracked how many times the arbiter had already intervened. A re-deadlock
re-invoked the arbiter without bound.
Add a persistent arbiter_intervention_count (new column + migration 020)
that survives the round-trip reset, and a configurable cap
ARBITER_MAX_INTERVENTIONS (default 1). Once the arbiter has intervened
that many times and the loop still deadlocks, requestArbiterOrEscalate
escalates straight to the user instead of re-invoking the arbiter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Merging origin/main collided on schema version 15: the deployment DB
recorded v15 as `reviewer_failure_count` (downstream-only), while upstream
defines v15 as `turn_progress_text`. The runner tracks applied migrations
by version number, so it skips upstream's `turn_progress_text` on the live
DB and never creates paired_turns.progress_text / progress_updated_at —
columns the merged runtime reads and writes (db/paired-turns.ts,
web-dashboard-data.ts), which would crash the live paired-room flow.
Add idempotent migration 019 to re-create those columns under a free
version number. No-op on fresh installs where migration 015 already ran.
Caught by a pre-flight boot of the merged code against a copy of the live DB.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
The Codex usage primer spawned a bare `codex` binary, but under bun/systemd
there is no global `codex` on PATH, so every slot failed with ENOENT and the
primer never ran. Resolve the vendored @openai/codex JS launcher and run it
through the current JS runtime (matching the codex-runner), with global-binary
and PATH fallbacks.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
The Codex-only shouldHoldFreshCodexUntilPrimer gate blocked fresh Codex
turns outside the dawn window and replied "Codex 사용량 정렬 대기 중...
다음 프라이머 시각 이후 다시 보내주세요", which Claude rooms never did.
The primer (usage-primer.ts) already fires unconditionally for both
providers at 08/13/18/23 KST, so this hold only delayed user turns
without affecting window alignment.
Remove the gate and its now-orphaned helpers (shouldHoldFreshCodexUntilPrimer,
CODEX_ALIGNMENT_AUTO_REPLY, msUntilNextPrimerSlotKST, formatPrimerHourKST,
and the getAllCodexAccounts import). Keep the 04-08 KST dawn alignment gap
block and the usage-exhaustion gate untouched.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
The previous fix gated the step_done deadlock guard on the same
deadlockThreshold every verdict uses — but handleReviewerCompletion
passes Infinity for that threshold when no arbiter is configured
(isArbiterEnabled()=false, e.g. production with the ARBITER__AGENT_TYPE
env typo). So step_done never tripped the guard and the
owner TASK_DONE ⇄ reviewer STEP_DONE loop still ran forever.
Give step_done its own always-finite threshold
(stepDoneDeadlockThreshold = ARBITER_DEADLOCK_THRESHOLD) independent of
arbiter availability. Once hit, requestArbiterOrEscalate routes to the
arbiter when enabled, or completes the task with completion_reason
'escalated' (user escalation) when not. Productive REVISE/continue
disagreement still uses deadlockThreshold and is intentionally left on
the existing policy (loops to owner when arbiter is off), matching the
existing "returns reviewer change requests to owner ... deadlock
threshold" test.
Adds a shared regression unit test (deadlockThreshold=Infinity +
step_done still escalates) and three integration tests through
completePairedExecutionContext: arbiter-off+threshold -> escalated,
arbiter-on+threshold -> arbiter_requested, below-threshold -> active.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
The reviewer step_done verdict mapped to request_owner_changes
unconditionally, bypassing the deadlock guard that every sibling
verdict respects. When the owner kept replying task_done and the
reviewer kept replying step_done, the room oscillated forever,
spamming duplicate reviewer messages into the channel.
Apply the same `roundTripCount >= deadlockThreshold` guard used by
the done_with_concerns/continue/default branch so step_done
oscillation escalates to the arbiter (or completes via escalation
when no arbiter) instead of looping. Reuses existing round_trip_count
and deadlockThreshold; no migration needed.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Per-turn agent runners do heavy file I/O (git clone, builds, ffmpeg/video
scratch) inside the ejclaw service cgroup. The kernel retains those reads as
reclaimable page cache, so systemd's MemoryCurrent climbs to 4-5GB even though
real anon usage stays ~300MB and the host has 13GB free. Add a soft MemoryHigh
so the cache is reclaimed under pressure before the number balloons; MemoryMax
stays unset (infinity) so a legitimate burst is throttled, never OOM-killed.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
editMessage is called repeatedly by the progress ticker and dashboard
refresh, so appending overflow chunks there duplicated text on every edit.
Revert editMessage to a pure single-message edit and instead expose the
platform per-message cap via Channel.maxMessageLength; deliverOpenWorkItem
now skips the doomed in-place edit and hands the full payload to the
chunking sendMessage path when the final exceeds that cap.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
- editMessage now splits >2000-char text: edits the tracked message with the
first chunk and appends the rest as follow-up messages, instead of throwing
Discord "Invalid Form Body … Must be 2000 or fewer" when promoting a long
owner result over a progress message.
- syncTrackedProgressMessage captures latestProgressText in a local before the
editMessage await, fixing a race where a concurrent resetProgressState()
nulled it and crashed with "null is not an object (this.latestProgressText.length)".
- Add editMessage regression tests (in-place edit vs. chunked overflow).
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
