Before, a Claude run that hit the account usage limit closed the open
paired task as claude_usage_limit, posted the reset time and consumed the
user's messages, so the work was lost and had to be re-requested.
Now the room is paused instead:
- The failed turn is released without touching the paired task (no
failure count, no status change, no follow-up) and the queue keeps the
human messages unconsumed, so the same turn can run again.
- The room is recorded in router_state `claude_usage_pause` with the reset
time (error text, else cached usage API reset, else a 10 min retry). A
banner time that just passed is read as "now", not tomorrow.
- While the limit lasts, paused rooms run nothing and any other room's
Claude turn pauses that room without spawning a run (the limit is
account-wide). A stop request still goes through and drops the room
from the resume list.
- The scheduler tick clears the pause 1 min after the reset, posts a
resume notice and re-queues the rooms; the pending turn or the kept
messages then run again. The state survives restarts.
Safety refusals keep closing the task as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deployment has a single Claude OAuth credential and a single Codex
ChatGPT account, so the N-account rotation machinery only added state
files, cooldown bookkeeping and code paths that never ran.
Removed:
- Claude token rotation (token-rotation.ts, token-rotation-base.ts):
rotate-on-limit, per-token cooldowns, token-rotation-state.json.
CLAUDE_CODE_OAUTH_TOKENS is still read, but only its first entry.
- Codex account rotation (codex-token-rotation.ts): account pool,
rotate-on-429/auth, dead-auth/cooldown tracking, round-robin advance,
d7 auto-rotate, codex-rotation-state.json, manual account switch.
- Account index plumbing in token-refresh, claude-credentials-path
(CLAUDE_ACCOUNTS_DIR / ~/.claude-accounts/<n>), Claude/Codex usage
collectors, the dashboard rows and the Codex warm-up.
- provider-retry.ts rotation loops and the Codex retry path.
Kept:
- Codex exclusive auth lease (one Codex run at a time), the 15-minute
waitForCodexAuthLease, and a usage-poll lease that skips while a run
holds the account (codex-auth.ts).
- Session auth copy and write-back of refreshed tokens
(syncCodexSessionAuthBack, syncClaudeSessionAuthBack).
- Codex auth path: ~/.codex-accounts/0/auth.json, else ~/.codex/auth.json
(now resolved per run, so a re-login needs no restart).
- Claude token refresh loop, CLAUDE_CREDENTIALS_PATH and the
CLAUDE_TOKEN_REFRESH_* / CLAUDE_USAGE_USE_HOME_CREDENTIALS flags; an
auth-expired run still force-refreshes the token and retries once.
- Error classification (classifyClaudeTrigger, classifyCodexTrigger).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No room has room_settings.is_main = 1, so every main-only path was dead,
and admin work is done from the host shell. Each room's agent may now
only act on its own room:
- IPC: messages/inbound injection only to the source room; schedule,
update, pause, resume and cancel only for the source room's tasks;
drop the assign_room and refresh_groups IPC tasks and the deps they
needed (assignRoom/syncGroups/getAvailableGroups/writeGroupsSnapshot)
- DB/host evidence is always scoped to the source group folder
- runner MCP server: drop the assign_room tool, the target_group_jid
parameters and the isMain flag; list_tasks shows only the room's tasks
- drop EJCLAW_IS_MAIN, RunnerInput.isMain and AgentInput.isMain; the
global CLAUDE.md memory is now included for every room
- drop the available_groups.json snapshot (available-groups.ts), the
inferred restart announcement that only went to the main room, and
the unused Channel.syncGroups hook
- setup register: drop --is-main; deregister-room: drop the main-room
refusal and --force, and fix its unknown[] binding type error
- remove the groups/main/ template
The room_settings.is_main column and migrations are untouched; new rows
get the column default (0) and nothing reads it anymore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scheduled tasks only suppressed phase 'progress', so intermediate
preambles (e.g. "I'll run the watchdog checks.") leaked to the chat
each run while the actual <internal>-wrapped result was correctly
stripped. The interactive path treats intermediate/tool-activity as
silent (toVisiblePhase); align the scheduler to forward only the final
message, with error outputs still falling through to rotation/error
handling.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Prevent duplicate CI completion notifications in paired rooms by checking
for existing active watchers with the same channel+provider+metadata before
creating a new one. Remove Task ID from watcher prompt construction to
avoid router secret redaction (task- IDs contain sk- pattern), passing
EJCLAW_RUNTIME_TASK_ID via env var instead and making cancel_task
auto-resolve when task_id is omitted.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace LLM-per-tick polling with direct `gh api` calls for GitHub
Actions watchers. New `ci_provider` discriminator column routes tasks
to either the host-driven path (zero LLM tokens) or the existing
generic LLM path. GitHub watchers poll at 15s intervals (min 10s)
via `checkGitHubActionsRun()` in the scheduler, with completion
messages sent directly to chat on terminal state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a Claude account is suspended, the API returns "Your organization does
not have access to Claude" on the success path or "Failed to authenticate.
API Error: 403 terminated" on the error path. Both are now classified as
'org-access-denied', suppressed from Discord output, and trigger automatic
token rotation. If all tokens are exhausted, enters cooldown without Kimi
fallback (same as usage-exhausted and auth-expired).
Changes:
- agent-error-detection: add isClaudeOrgAccessDeniedMessage(), expand
classifyClaudeAuthError() and shouldRotateClaudeToken()
- streamed-output-evaluator: detect org-access-denied in success-path chain
- provider-fallback: add NO_FALLBACK_COOLDOWN_REASONS set and
isPrimaryNoFallbackCooldownActive() helper
- provider-retry: handle org-access-denied in rotation loop
- message-agent-executor / task-scheduler: use generalized no-fallback
cooldown check
- Tests: +8 test cases across 5 test files (415 total passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix token rotation not taking effect: getCurrentToken() was shadowed
by static .env CLAUDE_CODE_OAUTH_TOKEN value in agent environment
- Don't fall back to Kimi on usage-exhausted (only on transient 429/network)
- Add disk cache for Claude usage API data (survives restarts, 429s)
- Rate-limit usage API calls to 1 per token per 5 minutes
- Add ignoreRateLimits option to rotateToken() for exhausted recovery
- Rotate to next token in getActiveProvider() when current is exhausted
- Add isUsageExhausted() helper to provider-fallback
Major reliability improvements to the message processing pipeline:
- Add messages.seq monotonic cursor replacing timestamp-based cursors,
preventing message loss from timestamp collisions with LIMIT queries
- Add work_items table separating agent production from delivery,
enabling delivery retry without re-running the agent
- Propagate Discord send failures instead of silently swallowing them
- Add Claude 429 → Kimi K2.5 automatic provider fallback with cooldown
- Fix follow-up turn state reset in live index.ts path (not just
message-runtime.ts) to prevent final output from being lost
- Add restart context tracking for graceful restart announcements
- Lazy migration from timestamp cursors to seq cursors for existing data
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add MEMENTO_MCP_SSE_URL/ACCESS_KEY/REMOTE_PATH to readEnvFile
- Merge .env vars into cleanEnv for runner process inheritance
- Claude Code runner: add memento-mcp via mcp-remote in mcpServers
- Codex runner: inject memento-mcp section into config.toml
- Various improvements: CI watch, task scheduler, DB, IPC auth
* fix: atomic claim prevents scheduled tasks from executing twice (#138)
Replace the two-phase getDueTasks() + deferred updateTaskAfterRun() with
an atomic SQLite transaction (claimDueTasks) that advances next_run
BEFORE dispatching tasks to the queue. This eliminates the race window
where subsequent scheduler polls re-discover in-progress tasks.
Key changes:
- claimDueTasks(): SELECT + UPDATE in a single db.transaction(), so no
poll can read stale next_run values. Once-tasks get next_run=NULL;
recurring tasks get next_run advanced to the future.
- computeNextRun(): anchors interval tasks to the scheduled time (not
Date.now()) to prevent cumulative drift. Includes a while-loop to
skip missed intervals and a guard against invalid interval values.
- updateTaskAfterRun(): simplified to only record last_run/last_result
since next_run is already handled by the claim.
Closes#138, #211, #300, #578
Co-authored-by: @taslim (PR #601)
Co-authored-by: @baijunjie (Issue #138)
Co-authored-by: @Michaelliv (Issue #300)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply prettier formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: track running task ID in GroupQueue to prevent duplicate execution (#138)
Previous commits implemented an "atomic claim" approach (claimDueTasks)
that advanced next_run before execution. Per Gavriel's review, this
solved the symptom at the wrong layer and introduced crash-recovery
risks for once-tasks.
This commit reverts claimDueTasks and instead fixes the actual bug:
GroupQueue.enqueueTask() only checked pendingTasks for duplicates, but
running tasks had already been shifted out. Adding runningTaskId to
GroupState closes that gap with a 3-line fix at the correct layer.
The computeNextRun() drift fix is retained, applied post-execution
where it belongs.
Closes#138, #211, #300, #578
Co-authored-by: @taslim (PR #601)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add changelog entry for scheduler duplicate fix
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add contributors for scheduler race condition fix
Co-Authored-By: Taslim <9999802+taslim@users.noreply.github.com>
Co-Authored-By: BaiJunjie <7956480+baijunjie@users.noreply.github.com>
Co-Authored-By: Michael <13676242+Michaelliv@users.noreply.github.com>
Co-Authored-By: Kyle Zhike Chen <3477852+kk17@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: gavrielc <gabicohen22@yahoo.com>
Co-authored-by: Taslim <9999802+taslim@users.noreply.github.com>
Co-authored-by: BaiJunjie <7956480+baijunjie@users.noreply.github.com>
Co-authored-by: Michael <13676242+Michaelliv@users.noreply.github.com>
Co-authored-by: Kyle Zhike Chen <3477852+kk17@users.noreply.github.com>