Root cause of recurring "Refresh token expired / invalid_grant" logouts: the
main refresh loop and each agent session's Claude CLI share one OAuth token
family but refresh independently. Anthropic rotates refresh tokens and revokes
the whole family if an already-rotated token is reused, so when a session
refreshed mid-turn the canonical copy went stale and its next refresh was
rejected — forcing a manual re-login.
Add syncClaudeSessionAuthBack (mirrors the existing Codex syncCodexSessionAuthBack):
after each Claude turn, if the session's CLAUDE_CONFIG_DIR credentials are
strictly newer than canonical (and same subscription), adopt them into the
canonical file. writeCredentials then fans the current token out to every
session dir, so no stale copy lingers to trigger family revocation. Decision
logic extracted to the pure, tested shouldAdoptSessionOAuth.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Deregistration only removed the base group folder, leaving the tribunal
reviewer/arbiter runtime behind: DB session rows keyed as "<folder>:reviewer"
/":arbiter" and on-disk dirs data/sessions/<folder>-reviewer/-arbiter (plus
ipc/workspaces variants). Include those role-suffixed variants in both the
sessions DELETE and the disk cleanup so a deregistered room leaves nothing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Instead of a fixed 60s interval from an arbitrary start offset, the base status
refresh now fires on each wall-clock minute boundary (:00), keeping the
minute-precision timestamp shown in the message accurate. Event-driven updates
(new message / agent activity) still refresh in between. Adds
msUntilNextMinuteBoundary with tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The retry refactor captured `const editMessage = channel.editMessage` and
called it detached, losing `this`. Every status edit then threw
"this.client is undefined", so each cycle failed all retries and reposted a
fresh (notifying) status message — ~50 reposts in 30 minutes. Bind the method
to the channel so the edit runs in place. Adds a regression test showing a
detached method fails while a bound one succeeds.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Raise the status dashboard base refresh from 10s to 60s, and refresh
immediately (debounced 1.5s) on events between ticks:
- a real chat message arrives in a registered room (index.ts onMessage)
- an agent starts/finishes a run, i.e. activity moves between rooms
(GroupQueue.setOnActivityChange fired on activeCount changes)
requestImmediateStatusUpdate() drives updateStatus out-of-band via a coalescing
trigger. The existing re-entrancy guard means the base periodic refresh does not
run while an edit-retry is in flight; a refresh requested during that window is
remembered and runs once afterward. Adds createCoalescingTrigger with tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 15s x2 edit-retry-before-repost policy is meant for transient blips during
steady-state operation, not the moment right after a (re)start. On the first
render after startup, use 0 retries: still edit the stored message in place if
possible, but if that edit fails, repost a fresh status message immediately
instead of waiting through the retry cycle. Subsequent ticks use the 2-retry
policy.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A transient Discord error (e.g. HTTP 503) on the periodic status-message edit
previously caused an immediate repost of a fresh status message. Now the edit
is retried up to 2 more times at 15s spacing, and a fresh message is only sent
if every attempt fails. Retry logic is extracted into the testable
editStatusMessageWithRetry helper (injectable sleep). Added a re-entrancy guard
so overlapping interval ticks are skipped while a slow retry runs, preventing
double-posts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When a room was unregistered first (room_settings row already gone), the
folder was unknown, so sessions rows and the on-disk group/workspace/session/
ipc folders (and any git worktree) were silently skipped — leaving orphans.
Back-trace the folder from the group_folder recorded on the chat's leftover
paired_tasks/work_items/scheduled_tasks/service_handoffs/paired_projects rows
so folder-scoped cleanup still runs. Targets now carry a folder list.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
task_run_logs.task_id references scheduled_tasks.id, not paired_tasks.id, so
the previous deletion (keyed by paired task ids) left orphaned run-log rows
behind — with foreign_keys OFF nothing cleaned them up. Delete task_run_logs
by the chat's scheduled_tasks ids before removing the scheduled_tasks rows.
Also auto-back up the DB to /home/claude/ejclaw-db-backup-<ts>.db before the
irreversible purge (skippable with --no-backup).
Verified end-to-end in a sandboxed DB copy: a seeded room's scheduled task +
3 run logs, paired data, work items, sessions, and router cursor are all
removed, disk folders deleted, while chats/messages and unrelated rooms stay
intact.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reusable script to fully unregister chat room(s) and purge all managed data
while keeping ONLY the chats channel row and messages history. Removes room
registration, paired tasks/turns/attempts/outputs/reservations/leases/
projects/handoffs, work items, scheduled tasks, sessions, router cursor, and
the on-disk group/workspace/session/ipc folders (git worktrees removed
cleanly). Supports --dry-run, refuses main rooms without --force, and skips
disk deletion for group folders still shared by another room.
Backs the "채팅 등록 해제" standing workflow (unregister + purge + restart).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fold in the pre-commit prettier reflow that the previous commit did not
restage. No behavior change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A room_settings row with a valid room_mode but an unrecognized mode_source
(e.g. the stray 'room' value that disabled the cgv-macro channel) was
silently dropped by getStoredRoomSettingsRowFromDatabase. That removed the
room from every binding lookup, so the router ignored the channel, it
vanished from the status list, and re-registration wedged on the
UNIQUE(chat_jid) constraint because assignRoom's "existing" probe uses the
same loader.
Coerce an invalid mode_source to 'explicit' (preserving the stored
room_mode) and log a warning so the corruption stays visible and self-heals
on the next assignRoom, instead of taking the channel silently offline.
room_mode is already protected by a column CHECK; mode_source was not.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rooms registered as tribunal had no work_dir and no paired_projects row, so
ensurePairedProject() returned null, no paired task was ever created, and only
the owner ran — the reviewer/arbiter never fired (web-vstock and 4 other rooms).
Add ensurePairedWorkspaceProvisioned(): defaults canonical work_dir to
groups/<folder>, guarantees it is a standalone git repo with an initial commit
(detected via a LOCAL .git so we never walk up into the EJClaw checkout and
create a stray worktree), and upserts the paired_projects row. Wire it into
setup/register.ts so every newly registered room is immediately usable by the
full owner→reviewer→arbiter flow. Adds a regression test.
turnController.finish() is the only path that clears the 5s progress ticker,
but it lives inside the caller's try block in message-runtime-turns.ts. When
runAgent throws / is aborted / is killed (e.g. a user "중단"), control jumps to
finally and finish() is skipped, leaving the ticker editing the Discord message
forever — an orphaned "stuck at 0s" zombie progress message with no backing
task/attempt.
Add an idempotent MessageTurnController.dispose() that tears down the progress
ticker + idle timer, and always call it from the caller's finally so timers are
cleared on every exit path. Adds a regression test.
Some Codex plans (e.g. Plus) return only a weekly window in `primary` with
`secondary: null`. applyCodexUsageToAccount dereferenced `secondary.usedPercent`
unconditionally, throwing a TypeError that refreshActiveCodexUsage swallowed at
debug level, so usage never applied and the dashboard row stayed blank (-1).
Guard the null secondary, and when a lone window is weekly (windowDurationMins
> 1440) route it into the 7d slot leaving 5h unknown. Accounts that report both
windows are unchanged. Adds a regression test for the secondary:null case.
Two causes of the paired room "keeps stopping" symptom:
- Reviewer approvals worded as "PROCEED" were parsed as 'continue'
(a change request), causing an owner TASK_DONE <-> reviewer PROCEED
ping-pong until the deadlock cap. parseReviewerVerdict() now treats a
leading PROCEED as approval so the turn finalizes after one round.
- When the Codex reviewer was unavailable, the owner's answer was held
for review and the user saw nothing. Now the held owner answer is
emitted with a "review skipped" notice on reviewer_codex_unavailable.
Verified: tsc --noEmit clean; 27 related vitest tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Claude usage poller retried /api/oauth/usage every 60s, but the endpoint
returns 429 with a longer Retry-After window (~93s). Retrying mid-cooldown
re-tripped the limit so the 429s never cleared and usage data never populated.
Record a cooldownUntil from the 429 Retry-After header (5min fallback when
absent) and skip the API until it closes; cleared on success. Dashboard now
shows a "429" indicator instead of a stale value when rate-limited.
Adds regression tests proving the cooldown outlasts the 60s throttle and
releases once the window passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
assign_room (MCP tool + host-side assignRoomInDatabase) and setup
register previously fell back to 'single' when no room_mode was given,
so newly added rooms silently lost the reviewer. Default to 'tribunal'
across the MCP zod schema, the IPC arg fallback, and the DB helper, and
update the ipc-auth expectation accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resolves the gitea/main <-> deployed-line merge:
- DB migrations: renumber gitea's colliding 019/020 to 021/022
(reviewer_failure_count -> v21, turn_progress_text_compat -> v22) so all
four migrations have distinct versions; update ordered list + bootstrap test.
- paired_tasks INSERT: add the missing VALUES placeholder so both new columns
(reviewer_failure_count + arbiter_intervention_count) bind (26 cols/values).
- index.ts: drop duplicate startUsagePrimer import from the auto-merge.
- discord output: keep the deployed pipeline (attachment rejection notice) and
call sanitizeForOutbound at the channel boundary so prose escaping done in
prepareDiscordOutbound is not double-applied; keep gitea's reviewer
silent-failure cap + router markdown-escape helpers.
- usage-primer/codex-warmup: keep the dawn-hold removal over gitea's primer.
Full test suite: only pre-existing env/bun-path failures remain; no merge regressions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reviewer/arbiter/owner Codex turns are no longer deferred during the
[03:00-08:00 KST] dawn window — they dispatch immediately at all hours.
Deletes codex-primer-alignment.ts and every hold call site (gating owner
hold + alignment notice, reviewer/arbiter queue hold, warm-up skip, primer
anchor-lock). The 08:00 primer still fires; it just no longer holds other
consumers. Tradeoff: the 13:00 KST 5h-reset anchoring is no longer enforced.
Also adds a user-visible next-step indicator after owner turns that do not
end the task (review_ready -> reviewer requested, arbiter_requested ->
arbiter called) so a same-looking status line is no longer ambiguous; no
extra line on completion to avoid duplicating the owner's final message.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Scheduled tasks only suppressed phase 'progress', so intermediate
preambles (e.g. "I'll run the watchdog checks.") leaked to the chat
each run while the actual <internal>-wrapped result was correctly
stripped. The interactive path treats intermediate/tool-activity as
silent (toVisiblePhase); align the scheduler to forward only the final
message, with error outputs still falling through to rotation/error
handling.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Show NVIDIA GPU utilization and VRAM used/total in the 서버 status block,
matching the existing CPU/Memory/Disk bar format. Gracefully omitted when
nvidia-smi is unavailable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The staging logic in 9c46cf6 copied any agent-declared file from outside the
room's allowed directories into a safe folder and attached it, which bypassed
the attachment directory allowlist (cross-room isolation / sensitive-file
protection). Removing it.
Kept: appendRejectionNotice / describeRejectedAttachments so rejected
attachments are surfaced in the visible message instead of being silently
dropped. This changes no security behavior — it only adds text when an
attachment was already going to be rejected.
Verified: outbound-attachments + final-delivery + discord tests 70/70, tsc clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Agent-generated files written to an arbitrary working path (e.g. TTS audio
under /home/claude/jarvis-tts) were rejected by validateOutboundAttachments as
"outside-allowed-dirs". The rejection was only logged; the MEDIA: directive had
already been stripped from the text, so the user got a message claiming a file
was attached with no file and no error.
- Stage attachments outside the room's allowed dirs into a safe per-group dir
(data/attachments/outbound/<group>) at the universal delivery choke point, so
the path delivery uses and revalidates is one the validator accepts. Files
already inside an allowed dir are untouched, preserving isolation checks.
- Surface any still-rejected attachment in the visible Discord body via
appendRejectionNotice, so delivery can never again silently drop a file.
Verified: outbound-attachments 22/22, discord 46/46 (incl. new integration test
asserting the notice lands in the sent body), final-delivery 5/5, tsc clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Address reviewer findings on the narrow dawn-hold approach:
- 08:00 release race (the daily failure): the dawn window's clock boundary
lapses at exactly 08:00, so a held reviewer turn released at 08:00:00 could
beat the primer to open the fresh window. Add an in-process anchor lock — the
08:00 primer sets a flag for the duration of its own Codex call, and a short
post-slot grace bridges the gap between the boundary lapse and the flag set.
Held consumers stay held until the primer's anchoring request completes.
- Owner gate now keys on the EFFECTIVE owner backend (configured codex owner OR
a global-failover override to Codex) via getEffectiveChannelLease, not the
static group.agentType — covers failover-to-Codex rooms.
- Dedupe the owner wait-notice to at most once per room per dawn window
(ALIGNMENT_NOTICE_KEY) instead of on every re-poll. Held turns still resume
exactly once: the gate returns without advancing the message cursor.
Note on coverage: the deep common boundary (runAgentForGroup) returns only
'success' | 'error' with no defer state — holding there would consume/lose the
message or trigger handoff/failover, so holds stay at the cursor-safe shallow
gates. Paired reviewer/arbiter turns (incl. scheduled review-ready
reconciliation) and codex-owner turns are covered; generic codex scheduled
tasks (runTask) and Claude->Codex handoffs at dawn remain rare residual edges.
Tests: anchor-lock + grace + slot/notice-key (codex-primer-alignment.test),
failover-owner hold + notice dedupe (message-runtime-gating.test). 27 focused
tests pass; the 9 pre-existing service-routing/migration failures predate this
work (verified at 728a8c2).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Codex 5h usage limit is a FIXED window (verified live: usedPercent climbs
while resetsAt stays put), so it can be anchored — but only if the 08:00 KST
primer is the first Codex request of a fresh window. Codex is otherwise used
near-continuously (reviewers/arbiters every paired room, codex-owner rooms,
warm-up), so 08:00 lands mid-window and the reset drifts (observed 15:47).
Restore the alignment hold from backup/primer-hold-work, but narrowed to a
single dawn window (03:00-08:00 KST) per user direction — the broad all-slots
version was previously reverted. One Codex window is at most 5h, so anything
opened before 03:00 has expired by 08:00, leaving the primer to anchor cleanly
=> 13:00 reset. Other slots (13/18/23) and all daytime hours are untouched.
- codex-primer-alignment.ts: narrow shouldHoldCodexForPrimerAlignment (03-08 KST)
- codex-warmup.ts: skip non-primer (non-forceAttempt) warm-ups during the hold
- message-runtime-queue.ts: defer Codex reviewer/arbiter turns during the hold
- message-runtime-gating.ts: defer fresh Codex-owner room turns during the hold
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Codex primer awaited refreshAllCodexAccountUsage + refreshActiveCodexUsage
before firing, purely to feed the old usage-based warm-up gating. With
forceAttempt the primer fires regardless of usage, so those pre-call refreshes
only added variable latency (the few-hundred-ms jitter that pushed the call off
the exact slot) and a needless dependency on the usage API. Fire first at the
slot; refresh once afterwards for observability (never delaying the call). The
hourly usage collector keeps cached usage current.
Verified: typecheck, build, bundle-smoke; usage-primer + codex-warmup suites pass
(primer now fires before any refresh; refreshAll no longer called).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The scheduled primer reused the opportunistic warm-up gates, so the 18:00 slot
(exactly 5h after 13:00, but firing a few hundred ms later) fell just under
minIntervalMs and was silently skipped as no_eligible_accounts — the primer did
not actually fire at every slot. Add a forceAttempt path: the fixed-slot primer
bypasses stagger, post-failure cooldown, min-interval, and per-account
usage/rate-limit filters, attempting a real Codex command on the active account
at every 08/13/18/23 slot and recording success/failure.
Reviewer/arbiter holding stays removed (per user). This does not pin the reset
clock — a trailing window still slides with later usage — but it guarantees the
"simple task at every fixed slot" the user asked for.
Verified: typecheck, build, bundle-smoke; new regression tests (13:00 success
then sub-5h 18:00 still fires; control without forceAttempt is skipped;
forceAttempt bypasses cooldown/stagger) + usage-primer/codex-warmup suites pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The user asked to NOT defer reviewer/arbiter Codex calls, but instead just fire
a simple primer task at each fixed slot independent of the reviewer. That is
already the live behavior (efc8c00: primer fires at 08/13/18/23 KST + the Claude
binary-path fix). Revert the hold/gating work (e324a1b..ef2b26e) so it can't be
deployed; it is preserved on branch backup/primer-hold-work in case the 08:00
overnight-exhaustion gap later warrants a minimal quiet window.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Address review gaps in the primer-alignment hold so the fixed-slot primer
reliably wins the first-request race for the shared Codex window:
- Extract the time-based hold into a dependency-free leaf module
(codex-primer-alignment.ts) so every Codex path can share it without import
cycles.
- Add a pre-slot hold window (2 min) in addition to the post-slot (5 min) and
the 04-08 KST dawn gap, closing the "request a beat before the slot" gap.
- Hold the dashboard's periodic Codex warm-up too: runCodexWarmupCycle now
skips during the hold unless called with isPrimer (the scheduled primer sets
it), so the warm-up can't anchor the window ahead of the primer.
- Remove the urgent @-mention bypass on the reviewer/arbiter and owner holds —
the reset is anchored unconditionally as requested.
- Reviewer/arbiter hold resumes exactly once after the window via the existing
per-revision claim (poll re-detect; no double run).
Note: the unified lease boundary (syncHostCodexSessionFiles) is synchronous and
throws a terminal "Codex unavailable", so it can't host a clean defer+resume;
gating stays at the async turn-scheduling layer that can re-queue. Honest caveat
unchanged: making the primer first is necessary but not sufficient to pin a
trailing reset.
Verified: typecheck, build, bundle-smoke; full suite 1482 passing with only the
9 pre-existing env-config baseline failures (service-routing/paired-context/
migrate-room owner=claude vs codex-main); new alignment + warmup-hold tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The usage primer fires at 08/13/18/23 KST but could not pin the Codex 5h
reset because reviewer/arbiter turns on the same shared account ran off-slot
and anchored the window first. Add a time-based hold
(shouldHoldCodexForPrimerAlignment) that keeps non-primer Codex turns quiet
during the 04-08 KST dawn gap and for a few minutes after each slot, so the
scheduled primer wins the first-request race. Applied at the reviewer/arbiter
dispatch site (the actual off-slot consumer here) and for codex-owner rooms;
urgent @-mention turns bypass it. Time-based by design — the usage/reset
figures are exactly the unreliable data, so a blunt clock rule is predictable
and testable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live evidence shows the Codex 5h limit is a trailing rolling window whose
reported reset slides forward with continued usage (observed 18:00 → 18:33
over ~33 min). A timed primer therefore cannot pin the reset to a fixed clock
time, and the previous "retry at reported reset" logic was both based on a
wrong fixed-window model and buggy under sliding resets (stale reset target,
single-shot). Revert it.
Keep the verified Claude primer binary-path fix (resolve repo-root
node_modules) and document the real rolling-window behavior so the primer is
correctly framed as a best-effort warm-up + success/failure record.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two evidence-based fixes for the usage-window primer that aligns the 5h
reset to fixed KST slots (08/13/18/23):
1. Claude primer always failed with binary_missing: it only looked under
runners/agent-runner/node_modules, but the SDK platform binary is hoisted
to the repo-root node_modules. Resolve both locations.
2. When a Codex slot lands while the 5h window is still active/exhausted, the
warm-up call fails with a rate-limit and the window only re-anchors at its
natural reset — leaving Codex un-primed until the next 5h slot. Parse the
account's known reset time and schedule a one-shot retry right after it so
the next window anchors close to the slot.
Also corrected a stale comment claiming a 04:00–08:00 gap is enforced by
message-runtime-gating (it is not).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migration 020 (arbiter_intervention_count) was added without updating the
hardcoded expected list in bootstrap.test.ts, breaking its schema-migration
assertions. Add the entry so the canonical `bun run test` suite passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The owner↔reviewer↔arbiter loop could repeat forever: when the arbiter
ruled PROCEED/REVISE/RESET it reset round_trip_count to 0, and nothing
tracked how many times the arbiter had already intervened. A re-deadlock
re-invoked the arbiter without bound.
Add a persistent arbiter_intervention_count (new column + migration 020)
that survives the round-trip reset, and a configurable cap
ARBITER_MAX_INTERVENTIONS (default 1). Once the arbiter has intervened
that many times and the loop still deadlocks, requestArbiterOrEscalate
escalates straight to the user instead of re-invoking the arbiter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Merging origin/main collided on schema version 15: the deployment DB
recorded v15 as `reviewer_failure_count` (downstream-only), while upstream
defines v15 as `turn_progress_text`. The runner tracks applied migrations
by version number, so it skips upstream's `turn_progress_text` on the live
DB and never creates paired_turns.progress_text / progress_updated_at —
columns the merged runtime reads and writes (db/paired-turns.ts,
web-dashboard-data.ts), which would crash the live paired-room flow.
Add idempotent migration 019 to re-create those columns under a free
version number. No-op on fresh installs where migration 015 already ran.
Caught by a pre-flight boot of the merged code against a copy of the live DB.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
The Codex usage primer spawned a bare `codex` binary, but under bun/systemd
there is no global `codex` on PATH, so every slot failed with ENOENT and the
primer never ran. Resolve the vendored @openai/codex JS launcher and run it
through the current JS runtime (matching the codex-runner), with global-binary
and PATH fallbacks.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
