feat: pause room work on the Claude usage limit and resume after reset

Before, a Claude run that hit the account usage limit closed the open
paired task as claude_usage_limit, posted the reset time and consumed the
user's messages, so the work was lost and had to be re-requested.

Now the room is paused instead:
- The failed turn is released without touching the paired task (no
  failure count, no status change, no follow-up) and the queue keeps the
  human messages unconsumed, so the same turn can run again.
- The room is recorded in router_state `claude_usage_pause` with the reset
  time (error text, else cached usage API reset, else a 10 min retry). A
  banner time that just passed is read as "now", not tomorrow.
- While the limit lasts, paused rooms run nothing and any other room's
  Claude turn pauses that room without spawning a run (the limit is
  account-wide). A stop request still goes through and drops the room
  from the resume list.
- The scheduler tick clears the pause 1 min after the reset, posts a
  resume notice and re-queues the rooms; the pending turn or the kept
  messages then run again. The state survives restarts.

Safety refusals keep closing the task as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codex
2026-10-13 13:14:56 +09:00
parent 1167023a13
commit aed7371cfa
19 changed files with 893 additions and 90 deletions

View File

@@ -13,7 +13,7 @@
- 마무리(finalize) 턴 비용 절감. (가) 리뷰어가 owner의 첫 TASK_DONE 답변을 첫 리뷰에서 바로 승인했고(리뷰 1회, owner 수정·중재 턴 없음) 승인 이후 작업 폴더가 그대로면 owner를 다시 실행하지 않고 그 답변(상태 줄 제외)을 최종 메시지로 올리고 task를 완료함. 검토 생략 답변에 "검토해줘"로 받은 리뷰가 바로 승인되면 답변을 반복하지 않고 `-# 검토 완료: …` 한 줄만 올림. (나) 그 밖의 마무리 턴은 task의 긴 작업 세션을 resume하지 않는 새 세션에서, 이번 task의 사용자 요청과 최근 turn 출력(최대 6개)과 마무리 형식 안내만 담은 프롬프트로 실행하고 세션을 저장하지 않음(방 메모리 브리핑도 생략). owner 규칙에 첫 TASK_DONE 답변도 최종 답변 형식을 따라야 한다고 명시 - 마무리(finalize) 턴 비용 절감. (가) 리뷰어가 owner의 첫 TASK_DONE 답변을 첫 리뷰에서 바로 승인했고(리뷰 1회, owner 수정·중재 턴 없음) 승인 이후 작업 폴더가 그대로면 owner를 다시 실행하지 않고 그 답변(상태 줄 제외)을 최종 메시지로 올리고 task를 완료함. 검토 생략 답변에 "검토해줘"로 받은 리뷰가 바로 승인되면 답변을 반복하지 않고 `-# 검토 완료: …` 한 줄만 올림. (나) 그 밖의 마무리 턴은 task의 긴 작업 세션을 resume하지 않는 새 세션에서, 이번 task의 사용자 요청과 최근 turn 출력(최대 6개)과 마무리 형식 안내만 담은 프롬프트로 실행하고 세션을 저장하지 않음(방 메모리 브리핑도 생략). owner 규칙에 첫 TASK_DONE 답변도 최종 답변 형식을 따라야 한다고 명시
- paired 방의 owner·리뷰어 세션을 paired task 단위로 새로 시작. 같은 task 안의 후속 턴(리뷰 반영, 같은 task에 들어온 사용자 후속 메시지)은 그 세션을 이어 쓰고, 새 task의 첫 턴은 이전 세션을 resume하지 않음. 세션이 어느 task 것인지는 `router_state`의 `paired_session_task:<세션 폴더>`에 기록. 이전 task의 마지막 최종 답변은 새 task의 turn 0(carry-forward)으로 계속 전달되며, 이를 위해 `PAIRED_CARRY_FORWARD_LATEST_OWNER_FINAL` 기본값을 `true`로 변경. single 방은 그대로 - paired 방의 owner·리뷰어 세션을 paired task 단위로 새로 시작. 같은 task 안의 후속 턴(리뷰 반영, 같은 task에 들어온 사용자 후속 메시지)은 그 세션을 이어 쓰고, 새 task의 첫 턴은 이전 세션을 resume하지 않음. 세션이 어느 task 것인지는 `router_state`의 `paired_session_task:<세션 폴더>`에 기록. 이전 task의 마지막 최종 답변은 새 task의 turn 0(carry-forward)으로 계속 전달되며, 이를 위해 `PAIRED_CARRY_FORWARD_LATEST_OWNER_FINAL` 기본값을 `true`로 변경. single 방은 그대로
- Claude가 사용 한도에 도달하면 Codex로 넘기는 대신 방에 "Claude 사용 한도에 도달했습니다. 21:40에 재개할 수 있습니다."처럼 재개 시각(KST)을 바로 알리고, 열린 paired task를 `claude_usage_limit`으로 닫음. 재개 시각은 오류 문구(`resets 9:40pm (Asia/Seoul)` 등)에서 읽고, 없으면 사용량 API 캐시에서, 그것도 없으면 "잠시 후 다시 시도해 주세요"로 안내. 조용한 자동 재시도는 하지 않음 - Claude가 사용 한도에 도달하면 Codex로 넘기지 않고 그 방의 작업을 일시정지했다가, 사용량이 초기화되면 자동으로 이어서 진행함. 한도에 걸린 turn은 실패로 세지 않고 paired task 상태를 그대로 두며, 처리하던 사용자 메시지도 처리 완료로 넘기지 않음. 방에는 "Claude 사용 한도에 도달해서 이 방의 작업을 일시정지했습니다. 21:40에 사용량이 초기화되면 자동으로 이어서 진행합니다."처럼 재개 시각(KST)을 알림. 한도는 계정 전체에 걸리므로 일시정지 중에는 다른 방의 Claude turn도 실행하지 않고 같은 방식으로 일시정지함(일시정지되지 않은 방의 Codex turn은 그대로 실행). 일시정지 상태는 `router_state`의 `claude_usage_pause`에 저장되어 재시작 후에도 유지되고, 스케줄러 tick이 초기화 시각 1분 뒤에 "Claude 사용량이 초기화되어서 일시정지했던 작업을 이어서 진행합니다."를 보내고 방들을 다시 큐에 넣음. 재개 시각은 오류 문구(`resets 9:40pm (Asia/Seoul)` 등)에서 읽고, 없으면 사용량 API 캐시에서, 그것도 없으면 10분 뒤에 다시 시도함. 일시정지 중 "중단"을 보내면 그 방의 작업을 버리고 재개 대상에서 뺌
- 기본 Claude 모델을 Claude Opus 5.5(`claude-opus-5-5`)로 상향. Opus 5.5는 Claude Code CLI 2.1.280 이상이 필요해 `@anthropic-ai/claude-agent-sdk`를 0.3.153 → 0.3.286(번들 CLI 2.1.286)으로 올림 (8b68885) - 기본 Claude 모델을 Claude Opus 5.5(`claude-opus-5-5`)로 상향. Opus 5.5는 Claude Code CLI 2.1.280 이상이 필요해 `@anthropic-ai/claude-agent-sdk`를 0.3.153 → 0.3.286(번들 CLI 2.1.286)으로 올림 (8b68885)
- `@openai/codex`를 0.133.0 → 0.159.3으로 상향. 구버전이 매 실행마다 내던 `failed to decode models response: unknown variant max` 오류와 로그 폭주가 사라지고, 단순 리뷰 턴이 49초 → 5초로 단축됨 (a7f2e02) - `@openai/codex`를 0.133.0 → 0.159.3으로 상향. 구버전이 매 실행마다 내던 `failed to decode models response: unknown variant max` 오류와 로그 폭주가 사라지고, 단순 리뷰 턴이 49초 → 5초로 단축됨 (a7f2e02)

View File

@@ -29,7 +29,7 @@ EJClaw는 Discord 위에서 동작하는 Tribunal 멀티에이전트 개발 보
- role-fixed Discord 봇 3개 체계 - role-fixed Discord 봇 3개 체계
- reviewer host runtime + read-only guard - reviewer host runtime + read-only guard
- 승인 후 변경 감지와 재리뷰 - 승인 후 변경 감지와 재리뷰
- Claude 사용 한도 도달 시 재개 시각을 방에 바로 안내 - Claude 사용 한도 도달 시 방 작업을 일시정지하고, 사용량이 초기화되면 자동으로 이어서 진행
- `room_settings` 기반 명시적 room assignment - `room_settings` 기반 명시적 room assignment
- 자연어 중단: 작업 중인 방에 "그만", "멈춰", "중단해", "취소", "stop" 같은 짧은 메시지만 보내면 실행 중인 에이전트를 끝내고 열린 paired task를 닫음 - 자연어 중단: 작업 중인 방에 "그만", "멈춰", "중단해", "취소", "stop" 같은 짧은 메시지만 보내면 실행 중인 에이전트를 끝내고 열린 paired task를 닫음
- Bun + SQLite 기반 빠른 런타임 - Bun + SQLite 기반 빠른 런타임

View File

@@ -74,7 +74,7 @@ ANTHROPIC_BASE_URL=
- `CLAUDE_CODE_OAUTH_TOKEN`: runner에 주입하는 Claude 토큰. 예전 `CLAUDE_CODE_OAUTH_TOKENS`(쉼표 구분)가 남아 있으면 첫 번째 값만 씁니다 - `CLAUDE_CODE_OAUTH_TOKEN`: runner에 주입하는 Claude 토큰. 예전 `CLAUDE_CODE_OAUTH_TOKENS`(쉼표 구분)가 남아 있으면 첫 번째 값만 씁니다
- `CLAUDE_CREDENTIALS_PATH`: 이 서비스의 Claude OAuth credentials 파일. `CLAUDE_TOKEN_REFRESH_USE_CREDENTIALS=true`면 만료 전에 자동 갱신하고 새 토큰을 `.env`에도 기록합니다 - `CLAUDE_CREDENTIALS_PATH`: 이 서비스의 Claude OAuth credentials 파일. `CLAUDE_TOKEN_REFRESH_USE_CREDENTIALS=true`면 만료 전에 자동 갱신하고 새 토큰을 `.env`에도 기록합니다
- Claude가 사용 한도에 도달하면 다른 provider로 넘기지 않습니다. 방에 `Claude 사용 한도에 도달했습니다. 21:40에 재개할 수 있습니다.`처럼 재개 시각을 보내고(시각을 모르면 `잠시 후 다시 시도해 주세요`), 열린 paired task를 `claude_usage_limit`으로 닫습니다. 다음 메시지부터 새 작업으로 처리됩니다 - Claude가 사용 한도에 도달하면 다른 provider로 넘기지 않고 그 방의 작업을 일시정지합니다. 방에 `Claude 사용 한도에 도달해서 이 방의 작업을 일시정지했습니다. 21:40에 사용량이 초기화되면 자동으로 이어서 진행합니다.`처럼 재개 시각을 보내고, paired task와 처리하던 메시지는 그대로 둡니다. 한도가 남아 있는 동안 다른 방의 Claude turn도 실행하지 않고 일시정지합니다. 초기화 시각 1분 뒤 스케줄러가 방들을 다시 큐에 넣어 같은 turn을 이어서 실행합니다. 재개 시각을 모르면 10분 뒤에 다시 시도합니다. 일시정지 상태는 `router_state.claude_usage_pause`에 저장되어 재시작 후에도 유지되며, 일시정지 중 "중단"을 보내면 그 방은 재개하지 않습니다
Codex 쪽은 **OAuth 세션 파일** 기준으로 동작합니다. `~/.codex-accounts/0/auth.json`을 쓰고, 없으면 `~/.codex/auth.json`을 씁니다. Codex 실행은 계정 잠금으로 한 번에 하나만 돌고, 잠겨 있으면 최대 15분 기다립니다. `OPENAI_API_KEY`를 Codex child process에 넘겨서 빌링하는 구조는 사용하지 않습니다. Codex 쪽은 **OAuth 세션 파일** 기준으로 동작합니다. `~/.codex-accounts/0/auth.json`을 쓰고, 없으면 `~/.codex/auth.json`을 씁니다. Codex 실행은 계정 잠금으로 한 번에 하나만 돌고, 잠겨 있으면 최대 15분 기다립니다. `OPENAI_API_KEY`를 Codex child process에 넘겨서 빌링하는 구조는 사용하지 않습니다.

View File

@@ -13,10 +13,10 @@ vi.mock('./claude-usage.js', () => ({
})); }));
import { import {
buildClaudeUsageLimitMessage, formatClaudeUsagePauseMessage,
formatClaudeUsageLimitMessage,
isClaudeUsageLimitReason, isClaudeUsageLimitReason,
parseClaudeLimitResetTime, parseClaudeLimitResetTime,
resolveClaudeUsageResume,
} from './claude-usage-limit.js'; } from './claude-usage-limit.js';
// 2026-10-01 19:00 KST // 2026-10-01 19:00 KST
@@ -67,39 +67,73 @@ describe('parseClaudeLimitResetTime', () => {
}); });
}); });
describe('Claude usage limit message', () => { describe('Claude usage limit pause', () => {
it('shows HH:mm for a reset later today and the date otherwise', () => { it('shows HH:mm for a reset later today and the date otherwise', () => {
expect( expect(
formatClaudeUsageLimitMessage(new Date('2026-10-01T12:40:00Z'), NOW), formatClaudeUsagePauseMessage(
).toBe('Claude 사용 한도에 도달했습니다. 21:40에 재개할 수 있습니다.'); { resumeAt: new Date('2026-10-01T12:40:00Z'), resetKnown: true },
expect( NOW,
formatClaudeUsageLimitMessage(new Date('2026-10-01T19:00:00Z'), NOW), ),
).toBe( ).toBe(
'Claude 사용 한도에 도달했습니다. 10월 2일 04:00에 재개할 수 있습니다.', 'Claude 사용 한도에 도달해서 이 방의 작업을 일시정지했습니다. 21:40에 사용량이 초기화되면 자동으로 이어서 진행합니다. 그 전에 멈추려면 "중단"이라고 보내 주세요.',
); );
expect(
formatClaudeUsagePauseMessage(
{ resumeAt: new Date('2026-10-01T19:00:00Z'), resetKnown: true },
NOW,
),
).toContain('10월 2일 04:00에 사용량이 초기화되면');
}); });
it('asks to retry later without a reset time', () => { it('says it will retry when the reset time is unknown', () => {
expect(formatClaudeUsageLimitMessage(null, NOW)).toBe( expect(
'Claude 사용 한도에 도달했습니다. 잠시 후 다시 시도해 주세요.', formatClaudeUsagePauseMessage(
); { resumeAt: new Date('2026-10-01T10:10:00Z'), resetKnown: false },
NOW,
),
).toContain('초기화 시각을 알 수 없어서 19:10에 다시 시도합니다.');
});
it('retries after the fallback delay without a reset time', () => {
expect(resolveClaudeUsageResume('API Error: 429', NOW)).toEqual({
resumeAt: new Date('2026-10-01T10:10:00Z'),
resetKnown: false,
});
}); });
it('falls back to the cached usage reset time', () => { it('falls back to the cached usage reset time', () => {
cachedResetAt.mockReturnValueOnce(new Date('2026-10-01T11:15:00Z')); cachedResetAt.mockReturnValueOnce(new Date('2026-10-01T11:15:00Z'));
expect(buildClaudeUsageLimitMessage('API Error: 429', NOW)).toBe( expect(resolveClaudeUsageResume('API Error: 429', NOW)).toEqual({
'Claude 사용 한도에 도달했습니다. 20:15에 재개할 수 있습니다.', resumeAt: new Date('2026-10-01T11:15:00Z'),
); resetKnown: true,
});
}); });
it('prefers the reset time in the error text', () => { it('prefers the reset time in the error text', () => {
cachedResetAt.mockReturnValueOnce(new Date('2026-10-01T11:15:00Z')); cachedResetAt.mockReturnValueOnce(new Date('2026-10-01T11:15:00Z'));
expect( expect(
buildClaudeUsageLimitMessage( resolveClaudeUsageResume(
"You've hit your limit · resets 9:40pm (Asia/Seoul)", "You've hit your limit · resets 9:40pm (Asia/Seoul)",
NOW, NOW,
), ),
).toBe('Claude 사용 한도에 도달했습니다. 21:40에 재개할 수 있습니다.'); ).toEqual({
resumeAt: new Date('2026-10-01T12:40:00Z'),
resetKnown: true,
});
});
it('does not push a banner time that just passed to tomorrow', () => {
// 21:41 KST, the banner still says 9:40pm: retry shortly, not in 24 hours.
const justAfter = Date.parse('2026-10-01T12:41:00Z');
expect(
resolveClaudeUsageResume(
"You've hit your limit · resets 9:40pm (Asia/Seoul)",
justAfter,
),
).toEqual({
resumeAt: new Date(justAfter + 10 * 60_000),
resetKnown: false,
});
}); });
it('treats only rate-limit and usage-exhausted triggers as usage limits', () => { it('treats only rate-limit and usage-exhausted triggers as usage limits', () => {

View File

@@ -1,17 +1,18 @@
/** /**
* User-facing notice for a Claude run that hit the account usage limit. * Reset time and user-facing notice for a Claude run that hit the account
* usage limit.
* *
* There is a single Claude account and no fallback provider, so the room is * There is a single Claude account and no fallback provider, so the room's
* told when Claude becomes available again instead of the work being handed * work is paused until the limit resets and then resumed (see
* off or retried silently. * claude-usage-pause.ts).
*/ */
import type { AgentTriggerReason } from './agent-error-detection.js'; import type { AgentTriggerReason } from './agent-error-detection.js';
import { getCachedClaudeUsageLimitResetAt } from './claude-usage.js'; import { getCachedClaudeUsageLimitResetAt } from './claude-usage.js';
import { TIMEZONE } from './config.js'; import { TIMEZONE } from './config.js';
const LIMIT_PREFIX = 'Claude 사용 한도에 도달했습니다.'; const PAUSE_PREFIX =
const RETRY_LATER = '잠시 후 다시 시도해 주세요.'; 'Claude 사용 한도에 도달해서 이 방의 작업을 일시정지했습니다.';
const MONTHS: Record<string, number> = { const MONTHS: Record<string, number> = {
jan: 1, jan: 1,
@@ -198,26 +199,48 @@ function formatResetTime(resetAt: Date, nowMs: number): string {
return sameDay ? time : `${reset.month}월 ${reset.day}일 ${time}`; return sameDay ? time : `${reset.month}월 ${reset.day}일 ${time}`;
} }
export function formatClaudeUsageLimitMessage( /** A banner time that passed less than this long ago means "resetting now", not tomorrow. */
resetAt: Date | null, const JUST_PASSED_RESET_GRACE_MS = 15 * 60_000;
nowMs: number = Date.now(), /** Retry delay when the limit error names no usable reset time. */
): string { export const CLAUDE_USAGE_RETRY_FALLBACK_MS = 10 * 60_000;
if (!resetAt || resetAt.getTime() <= nowMs) {
return `${LIMIT_PREFIX} ${RETRY_LATER}`; export interface ClaudeUsageResume {
} resumeAt: Date;
return `${LIMIT_PREFIX} ${formatResetTime(resetAt, nowMs)}에 재개할 수 있습니다.`; /** False when no reset time was found and resumeAt is only a retry time. */
resetKnown: boolean;
} }
/** /**
* Notice for the room: the reset time comes from the error text, else from * When paused Claude work may resume: the reset time in the error text, else
* the cached usage API data, else the user is asked to retry later. * the cached usage API reset, else a short retry delay.
*
* A banner such as "resets 9:40pm" read at 21:41 is treated as "now" instead
* of tomorrow 21:40, so a limit that outlives its reset by a minute is not
* paused for a whole day.
*/ */
export function buildClaudeUsageLimitMessage( export function resolveClaudeUsageResume(
errorText: string | null | undefined, errorText: string | null | undefined,
nowMs: number = Date.now(), nowMs: number = Date.now(),
): string { ): ClaudeUsageResume {
const resetAt = const resetAt =
parseClaudeLimitResetTime(errorText, nowMs) ?? parseClaudeLimitResetTime(errorText, nowMs - JUST_PASSED_RESET_GRACE_MS) ??
getCachedClaudeUsageLimitResetAt(nowMs); getCachedClaudeUsageLimitResetAt(nowMs);
return formatClaudeUsageLimitMessage(resetAt, nowMs); if (resetAt && resetAt.getTime() > nowMs) {
return { resumeAt: resetAt, resetKnown: true };
}
return {
resumeAt: new Date(nowMs + CLAUDE_USAGE_RETRY_FALLBACK_MS),
resetKnown: false,
};
}
export function formatClaudeUsagePauseMessage(
resume: ClaudeUsageResume,
nowMs: number = Date.now(),
): string {
const at = formatResetTime(resume.resumeAt, nowMs);
const when = resume.resetKnown
? `${at}에 사용량이 초기화되면 자동으로 이어서 진행합니다.`
: `초기화 시각을 알 수 없어서 ${at}에 다시 시도합니다.`;
return `${PAUSE_PREFIX} ${when} 그 전에 멈추려면 "중단"이라고 보내 주세요.`;
} }

View File

@@ -0,0 +1,140 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
const { routerState } = vi.hoisted(() => ({
routerState: new Map<string, string>(),
}));
vi.mock('./db.js', () => ({
getRouterState: (key: string) => routerState.get(key),
setRouterState: (key: string, value: string) => {
routerState.set(key, value);
},
}));
vi.mock('./logger.js', () => ({
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() },
}));
import {
CLAUDE_USAGE_PAUSE_KEY,
CLAUDE_USAGE_RESUME_MARGIN_MS,
consumeRunPausedForClaudeUsage,
getActiveClaudeUsagePause,
isRoomPausedForClaudeUsage,
markRunPausedForClaudeUsage,
pauseRoomForClaudeUsage,
releaseRoomFromClaudeUsagePause,
takeRoomsReadyToResume,
} from './claude-usage-pause.js';
import { isSupportedRouterStateKey } from './db/router-state.js';
const NOW = new Date('2026-10-13T03:00:00Z');
const RESET = new Date('2026-10-13T05:00:00Z');
beforeEach(() => {
routerState.clear();
});
describe('Claude usage pause', () => {
it('uses a router_state key that startup accepts', () => {
expect(isSupportedRouterStateKey(CLAUDE_USAGE_PAUSE_KEY)).toBe(true);
});
it('pauses rooms until the reset and keeps the later reset time', () => {
pauseRoomForClaudeUsage({
chatJid: 'room-a',
resumeAt: RESET,
resetKnown: true,
now: NOW,
});
const pause = pauseRoomForClaudeUsage({
chatJid: 'room-b',
resumeAt: new Date(NOW.getTime() + 10 * 60_000),
resetKnown: false,
now: NOW,
});
expect(pause).toEqual({ until: RESET, resetKnown: true });
expect(isRoomPausedForClaudeUsage('room-a')).toBe(true);
expect(isRoomPausedForClaudeUsage('room-b')).toBe(true);
expect(isRoomPausedForClaudeUsage('room-c')).toBe(false);
expect(getActiveClaudeUsagePause(NOW.getTime())).toEqual({
until: RESET,
resetKnown: true,
});
});
it('moves the pause later when a later reset is reported', () => {
pauseRoomForClaudeUsage({
chatJid: 'room-a',
resumeAt: RESET,
resetKnown: false,
now: NOW,
});
const later = new Date(RESET.getTime() + 60 * 60_000);
expect(
pauseRoomForClaudeUsage({
chatJid: 'room-a',
resumeAt: later,
resetKnown: true,
now: NOW,
}),
).toEqual({ until: later, resetKnown: true });
});
it('holds the rooms until the reset plus a margin, then hands them back once', () => {
pauseRoomForClaudeUsage({
chatJid: 'room-a',
resumeAt: RESET,
resetKnown: true,
now: NOW,
});
pauseRoomForClaudeUsage({
chatJid: 'room-b',
resumeAt: RESET,
resetKnown: true,
now: NOW,
});
expect(takeRoomsReadyToResume(RESET.getTime())).toEqual([]);
expect(getActiveClaudeUsagePause(RESET.getTime())).toBeNull();
expect(isRoomPausedForClaudeUsage('room-a')).toBe(true);
const resumeAt = RESET.getTime() + CLAUDE_USAGE_RESUME_MARGIN_MS;
expect(takeRoomsReadyToResume(resumeAt)).toEqual(['room-a', 'room-b']);
expect(isRoomPausedForClaudeUsage('room-a')).toBe(false);
expect(takeRoomsReadyToResume(resumeAt)).toEqual([]);
});
it('releases one room when its work is stopped', () => {
pauseRoomForClaudeUsage({
chatJid: 'room-a',
resumeAt: RESET,
resetKnown: true,
now: NOW,
});
pauseRoomForClaudeUsage({
chatJid: 'room-b',
resumeAt: RESET,
resetKnown: true,
now: NOW,
});
expect(releaseRoomFromClaudeUsagePause('room-a')).toBe(true);
expect(releaseRoomFromClaudeUsagePause('room-a')).toBe(false);
expect(isRoomPausedForClaudeUsage('room-a')).toBe(false);
expect(isRoomPausedForClaudeUsage('room-b')).toBe(true);
});
it('ignores an unreadable stored state', () => {
routerState.set(CLAUDE_USAGE_PAUSE_KEY, '{not json');
expect(getActiveClaudeUsagePause(NOW.getTime())).toBeNull();
expect(isRoomPausedForClaudeUsage('room-a')).toBe(false);
expect(takeRoomsReadyToResume(NOW.getTime())).toEqual([]);
});
it('reports a paused run once', () => {
markRunPausedForClaudeUsage('run-1');
expect(consumeRunPausedForClaudeUsage('run-1')).toBe(true);
expect(consumeRunPausedForClaudeUsage('run-1')).toBe(false);
expect(consumeRunPausedForClaudeUsage('run-2')).toBe(false);
});
});

139
src/claude-usage-pause.ts Normal file
View File

@@ -0,0 +1,139 @@
/**
* Rooms whose Claude work is paused because the Claude account hit its usage
* limit.
*
* There is one Claude account, so the limit is global: while it lasts, every
* Claude run would fail the same way. The first run that hits it (or that
* would start while it lasts) pauses its room: the paired task is left as it
* was, the human messages stay unconsumed, and the room is recorded here.
* Paused rooms run nothing until the reset time; the scheduler tick then
* clears the pause and re-queues them, so the interrupted turn runs again.
*
* Stored in router_state so a restart keeps the pause and still resumes.
*/
import { getRouterState, setRouterState } from './db.js';
import { logger } from './logger.js';
export const CLAUDE_USAGE_PAUSE_KEY = 'claude_usage_pause';
/** Wait this long past the reset time before resuming, to not race it. */
export const CLAUDE_USAGE_RESUME_MARGIN_MS = 60_000;
export const CLAUDE_USAGE_RESUME_NOTICE =
'Claude 사용량이 초기화되어서 일시정지했던 작업을 이어서 진행합니다.';
interface ClaudeUsagePauseState {
/** ISO time the limit resets (or the next retry when unknown). */
until: string;
resetKnown: boolean;
/** chatJid -> ISO time the room was paused. */
rooms: Record<string, string>;
}
export interface ActiveClaudeUsagePause {
until: Date;
resetKnown: boolean;
}
function readState(): ClaudeUsagePauseState | null {
const raw = getRouterState(CLAUDE_USAGE_PAUSE_KEY);
if (!raw) return null;
try {
const parsed = JSON.parse(raw) as ClaudeUsagePauseState | null;
if (!parsed || !Number.isFinite(Date.parse(parsed.until))) return null;
return { ...parsed, rooms: parsed.rooms ?? {} };
} catch {
logger.warn({ raw }, 'Ignoring unreadable Claude usage pause state');
return null;
}
}
function writeState(state: ClaudeUsagePauseState | null): void {
setRouterState(CLAUDE_USAGE_PAUSE_KEY, JSON.stringify(state));
}
/**
* Pause `chatJid` until `resumeAt` (kept if a later pause already exists).
* Returns the pause now in effect.
*/
export function pauseRoomForClaudeUsage(args: {
chatJid: string;
resumeAt: Date;
resetKnown: boolean;
now?: Date;
}): ActiveClaudeUsagePause {
const now = args.now ?? new Date();
const state = readState();
const keepExisting =
state !== null && Date.parse(state.until) >= args.resumeAt.getTime();
const next: ClaudeUsagePauseState = {
until: keepExisting ? state.until : args.resumeAt.toISOString(),
resetKnown: keepExisting ? state.resetKnown : args.resetKnown,
rooms: { ...(state?.rooms ?? {}) },
};
next.rooms[args.chatJid] ??= now.toISOString();
writeState(next);
logger.warn(
{ chatJid: args.chatJid, until: next.until, resetKnown: next.resetKnown },
'Paused room work until the Claude usage limit resets',
);
return { until: new Date(next.until), resetKnown: next.resetKnown };
}
/** The Claude usage limit still in effect, or null. */
export function getActiveClaudeUsagePause(
nowMs: number = Date.now(),
): ActiveClaudeUsagePause | null {
const state = readState();
if (!state) return null;
const until = new Date(state.until);
return until.getTime() > nowMs
? { until, resetKnown: state.resetKnown }
: null;
}
export function isRoomPausedForClaudeUsage(chatJid: string): boolean {
return Boolean(readState()?.rooms[chatJid]);
}
/** Drop a room from the pause (the user stopped its work). */
export function releaseRoomFromClaudeUsagePause(chatJid: string): boolean {
const state = readState();
if (!state?.rooms[chatJid]) return false;
const rooms = { ...state.rooms };
delete rooms[chatJid];
writeState({ ...state, rooms });
return true;
}
/**
* Once the reset time (plus a margin) has passed, clear the pause and return
* the rooms to resume. Returns [] while the pause still holds.
*/
export function takeRoomsReadyToResume(nowMs: number = Date.now()): string[] {
const state = readState();
if (!state) return [];
if (Date.parse(state.until) + CLAUDE_USAGE_RESUME_MARGIN_MS > nowMs) {
return [];
}
writeState(null);
return Object.keys(state.rooms);
}
// Runs that ended paused, so the queue keeps their messages unconsumed.
const pausedRunIds = new Set<string>();
const MAX_TRACKED_PAUSED_RUNS = 100;
export function markRunPausedForClaudeUsage(runId: string): void {
pausedRunIds.add(runId);
if (pausedRunIds.size > MAX_TRACKED_PAUSED_RUNS) {
const oldest = pausedRunIds.values().next().value;
if (oldest !== undefined) pausedRunIds.delete(oldest);
}
}
/** True once for a run that ended paused by the Claude usage limit. */
export function consumeRunPausedForClaudeUsage(runId: string): boolean {
return pausedRunIds.delete(runId);
}

View File

@@ -36,7 +36,12 @@ export function getRouterStateFromDatabase(
* the same list, so an unregistered key fails on its first write and in tests * the same list, so an unregistered key fails on its first write and in tests
* instead of at the next startup. * instead of at the next startup.
*/ */
const SUPPORTED_ROUTER_STATE_KEYS = new Set(['last_seq', 'last_agent_seq']); const SUPPORTED_ROUTER_STATE_KEYS = new Set([
'last_seq',
'last_agent_seq',
// claude-usage-pause.ts
'claude_usage_pause',
]);
export const PAIRED_SESSION_TASK_KEY_PREFIX = 'paired_session_task:'; export const PAIRED_SESSION_TASK_KEY_PREFIX = 'paired_session_task:';
const SUPPORTED_ROUTER_STATE_KEY_PREFIXES = [PAIRED_SESSION_TASK_KEY_PREFIX]; const SUPPORTED_ROUTER_STATE_KEY_PREFIXES = [PAIRED_SESSION_TASK_KEY_PREFIX];

View File

@@ -44,17 +44,18 @@ function releaseInterruptedPairedExecution(
taskId: string, taskId: string,
runId: string, runId: string,
log: ExecutorLog, log: ExecutorLog,
cause = 'a human message interrupted the turn',
): void { ): void {
log.info( log.info(
{ pairedTaskId: taskId, runId }, { pairedTaskId: taskId, runId },
'Released paired execution lease without counting a failure because a human message interrupted the turn', `Released paired execution lease without counting a failure because ${cause}`,
); );
try { try {
releasePairedTaskExecutionLease({ taskId, runId }); releasePairedTaskExecutionLease({ taskId, runId });
} catch (err) { } catch (err) {
log.warn( log.warn(
{ pairedTaskId: taskId, runId, err }, { pairedTaskId: taskId, runId, err, cause },
'Failed to release paired execution lease after human interruption', 'Failed to release paired execution lease without counting a failure',
); );
} }
} }
@@ -178,6 +179,11 @@ export interface PairedExecutionLifecycle {
/** The text to deliver for an accepted final output. */ /** The text to deliver for an accepted final output. */
resolveFinalDeliveryText(outputText: string): string; resolveFinalDeliveryText(outputText: string): string;
completeImmediately(args: { status: 'succeeded' | 'failed' }): void; completeImmediately(args: { status: 'succeeded' | 'failed' }): void;
/**
* End the turn without touching the paired task (no failure count, no
* status change, no follow-up) so the same turn can run again later.
*/
releaseForClaudeUsagePause(): void;
markStatus(status: 'succeeded' | 'failed'): void; markStatus(status: 'succeeded' | 'failed'): void;
markSawOutput(sawOutput: boolean): void; markSawOutput(sawOutput: boolean): void;
getSummary(): string | null; getSummary(): string | null;
@@ -208,6 +214,7 @@ interface FinalizeState {
class PairedExecutionLifecycleController implements PairedExecutionLifecycle { class PairedExecutionLifecycleController implements PairedExecutionLifecycle {
private pairedExecutionStatus: PairedExecutionStatus = 'failed'; private pairedExecutionStatus: PairedExecutionStatus = 'failed';
private pausedForClaudeUsage = false;
private pairedExecutionSummary: string | null = null; private pairedExecutionSummary: string | null = null;
private pairedFinalOutput: string | null = null; private pairedFinalOutput: string | null = null;
private pairedFinalAttachments: OutboundAttachment[] = []; private pairedFinalAttachments: OutboundAttachment[] = [];
@@ -330,6 +337,23 @@ class PairedExecutionLifecycleController implements PairedExecutionLifecycle {
this.pairedExecutionCompleted = true; this.pairedExecutionCompleted = true;
} }
releaseForClaudeUsagePause(): void {
const { pairedExecutionContext, runId, log } = this.args;
this.pausedForClaudeUsage = true;
this.pairedExecutionStatus = 'failed';
if (!pairedExecutionContext || this.pairedExecutionCompleted) {
return;
}
this.clearLeaseHeartbeat();
releaseInterruptedPairedExecution(
pairedExecutionContext.task.id,
runId,
log,
'Claude hit its usage limit and the room is paused',
);
this.pairedExecutionCompleted = true;
}
markStatus(status: PairedExecutionStatus): void { markStatus(status: PairedExecutionStatus): void {
this.pairedExecutionStatus = status; this.pairedExecutionStatus = status;
} }
@@ -680,7 +704,11 @@ class PairedExecutionLifecycleController implements PairedExecutionLifecycle {
): Promise<void> { ): Promise<void> {
const { chatJid, completedRole, onOutput, pairedExecutionContext } = const { chatJid, completedRole, onOutput, pairedExecutionContext } =
this.args; this.args;
if (!pairedExecutionContext || state.interruptedByHumanMessage) { if (
!pairedExecutionContext ||
state.interruptedByHumanMessage ||
this.pausedForClaudeUsage
) {
return; return;
} }
const finishedTask = getPairedTaskById(pairedExecutionContext.task.id); const finishedTask = getPairedTaskById(pairedExecutionContext.task.id);

View File

@@ -225,6 +225,7 @@ import * as sessionRecovery from './session-recovery.js';
import * as serviceRouting from './service-routing.js'; import * as serviceRouting from './service-routing.js';
import * as tokenRefresh from './token-refresh.js'; import * as tokenRefresh from './token-refresh.js';
import * as claudeOauthToken from './claude-oauth-token.js'; import * as claudeOauthToken from './claude-oauth-token.js';
import { consumeRunPausedForClaudeUsage } from './claude-usage-pause.js';
import type { RegisteredGroup } from './types.js'; import type { RegisteredGroup } from './types.js';
function makeGroup(): RegisteredGroup { function makeGroup(): RegisteredGroup {
@@ -2754,8 +2755,23 @@ describe('runAgentForGroup room memory', () => {
}); });
describe('runAgentForGroup Claude failure recovery', () => { describe('runAgentForGroup Claude failure recovery', () => {
let routerState: Map<string, string>;
const readUsagePause = () =>
JSON.parse(routerState.get('claude_usage_pause') ?? 'null') as {
until: string;
resetKnown: boolean;
rooms: Record<string, string>;
} | null;
beforeEach(() => { beforeEach(() => {
vi.resetAllMocks(); vi.resetAllMocks();
routerState = new Map();
vi.mocked(db.getRouterState).mockImplementation((key) =>
routerState.get(key),
);
vi.mocked(db.setRouterState).mockImplementation((key, value) => {
routerState.set(key, value);
});
resetPairedFollowUpScheduleState(); resetPairedFollowUpScheduleState();
vi.mocked(buildRoomMemoryBriefing).mockResolvedValue(undefined); vi.mocked(buildRoomMemoryBriefing).mockResolvedValue(undefined);
vi.mocked(claudeOauthToken.getCurrentClaudeToken).mockReturnValue( vi.mocked(claudeOauthToken.getCurrentClaudeToken).mockReturnValue(
@@ -3039,7 +3055,7 @@ describe('runAgentForGroup Claude failure recovery', () => {
expect(outputs).toEqual([]); expect(outputs).toEqual([]);
}); });
it('tells the room when Claude resets instead of retrying on usage exhaustion', async () => { it('pauses the room until Claude resets instead of retrying on usage exhaustion', async () => {
const outputs: string[] = []; const outputs: string[] = [];
vi.mocked(agentRunner.runAgentProcess).mockImplementationOnce( vi.mocked(agentRunner.runAgentProcess).mockImplementationOnce(
@@ -3077,11 +3093,17 @@ describe('runAgentForGroup Claude failure recovery', () => {
expect(agentRunner.runAgentProcess).toHaveBeenCalledTimes(1); expect(agentRunner.runAgentProcess).toHaveBeenCalledTimes(1);
expect(outputs).toHaveLength(1); expect(outputs).toHaveLength(1);
expect(outputs[0]).toMatch( expect(outputs[0]).toMatch(
/^Claude 사용 한도에 도달했습니다\. (\d+월 \d+일 )?04:00에 재개할 수 있습니다\.$/, /^Claude 사용 한도에 도달해서 이 방의 작업을 일시정지했습니다\. (\d+월 \d+일 )?04:00에 사용량이 초기화되면 자동으로 이어서 진행합니다\./,
);
const pause = readUsagePause();
expect(pause?.resetKnown).toBe(true);
expect(Object.keys(pause?.rooms ?? {})).toEqual(['group@test']);
expect(consumeRunPausedForClaudeUsage('run-all-exhausted-error')).toBe(
true,
); );
}); });
it('closes the paired task when the Claude owner hits its usage limit', async () => { it('keeps the paired task as it was when the Claude owner hits its usage limit', async () => {
const outputs: string[] = []; const outputs: string[] = [];
const pairedTask = { const pairedTask = {
id: 'paired-task-usage-limit', id: 'paired-task-usage-limit',
@@ -3137,25 +3159,87 @@ describe('runAgentForGroup Claude failure recovery', () => {
expect(result).toBe('success'); expect(result).toBe('success');
expect(agentRunner.runAgentProcess).toHaveBeenCalledTimes(1); expect(agentRunner.runAgentProcess).toHaveBeenCalledTimes(1);
expect(db.updatePairedTaskIfUnchanged).toHaveBeenCalledWith( // No status change and no failure count: the same turn runs on resume.
'paired-task-usage-limit', expect(db.updatePairedTaskIfUnchanged).not.toHaveBeenCalled();
'2026-04-06T00:00:00.000Z',
expect.objectContaining({
status: 'completed',
completion_reason: 'claude_usage_limit',
}),
);
expect( expect(
pairedExecutionContext.completePairedExecutionContext, pairedExecutionContext.completePairedExecutionContext,
).toHaveBeenCalledWith( ).not.toHaveBeenCalled();
expect.objectContaining({ expect(db.releasePairedTaskExecutionLease).toHaveBeenCalledWith({
taskId: 'paired-task-usage-limit', taskId: 'paired-task-usage-limit',
status: 'failed', runId: 'run-paired-usage-limit',
});
expect(outputs).toEqual([
expect.stringMatching(
/^Claude 사용 한도에 도달해서 이 방의 작업을 일시정지했습니다\. .*다시 시도합니다\./,
),
]);
expect(readUsagePause()?.rooms).toHaveProperty('group@test');
expect(consumeRunPausedForClaudeUsage('run-paired-usage-limit')).toBe(true);
});
it('pauses a Claude turn without running it while the usage limit lasts', async () => {
const outputs: string[] = [];
const until = new Date(Date.now() + 60 * 60_000);
routerState.set(
'claude_usage_pause',
JSON.stringify({
until: until.toISOString(),
resetKnown: true,
rooms: { 'other@test': new Date().toISOString() },
}), }),
); );
const result = await runAgentForGroup(makeDeps(), {
group: makeGroup(),
prompt: 'hello',
chatJid: 'group@test',
runId: 'run-during-usage-pause',
onOutput: async (output) => {
const text =
output.output && 'text' in output.output
? output.output.text
: output.result;
if (typeof text === 'string') outputs.push(text);
},
});
expect(result).toBe('success');
expect(agentRunner.runAgentProcess).not.toHaveBeenCalled();
expect(outputs).toEqual([ expect(outputs).toEqual([
expect.stringMatching(/^Claude 사용 한도에 도달했습니다\. /), expect.stringMatching(/사용량이 초기화되면 자동으로 이어서 진행합니다/),
]); ]);
const pause = readUsagePause();
expect(pause?.until).toBe(until.toISOString());
expect(Object.keys(pause?.rooms ?? {}).sort()).toEqual([
'group@test',
'other@test',
]);
expect(consumeRunPausedForClaudeUsage('run-during-usage-pause')).toBe(true);
});
it('runs a Claude turn normally once the usage limit has reset', async () => {
routerState.set(
'claude_usage_pause',
JSON.stringify({
until: new Date(Date.now() - 1_000).toISOString(),
resetKnown: true,
rooms: { 'group@test': new Date().toISOString() },
}),
);
vi.mocked(agentRunner.runAgentProcess).mockResolvedValueOnce({
status: 'success',
result: 'done',
});
await runAgentForGroup(makeDeps(), {
group: makeGroup(),
prompt: 'hello',
chatJid: 'group@test',
runId: 'run-after-usage-reset',
});
expect(agentRunner.runAgentProcess).toHaveBeenCalledTimes(1);
expect(consumeRunPausedForClaudeUsage('run-after-usage-reset')).toBe(false);
}); });
it('closes the task, clears the session and posts a notice on a safeguards refusal', async () => { it('closes the task, clears the session and posts a notice on a safeguards refusal', async () => {
@@ -3420,7 +3504,9 @@ describe('runAgentForGroup Claude failure recovery', () => {
expect(result).toBe('success'); expect(result).toBe('success');
expect(outputs).toEqual([ expect(outputs).toEqual([
'대화 요약 중...', '대화 요약 중...',
expect.stringMatching(/^Claude 사용 한도에 도달했습니다\. .*02:00에/), expect.stringMatching(
/^Claude 사용 한도에 도달해서 이 방의 작업을 일시정지했습니다\. .*02:00에/,
),
]); ]);
}); });

View File

@@ -9,9 +9,16 @@ import type { PairedTurnIdentity } from './paired-turn-identity.js';
import { executeMessageAgentAttemptLifecycle } from './message-agent-executor-lifecycle.js'; import { executeMessageAgentAttemptLifecycle } from './message-agent-executor-lifecycle.js';
import { runMessageAgentAttempt } from './message-agent-executor-attempt-runner.js'; import { runMessageAgentAttempt } from './message-agent-executor-attempt-runner.js';
import { import {
buildClaudeUsageLimitMessage, formatClaudeUsagePauseMessage,
isClaudeUsageLimitReason, isClaudeUsageLimitReason,
resolveClaudeUsageResume,
type ClaudeUsageResume,
} from './claude-usage-limit.js'; } from './claude-usage-limit.js';
import {
getActiveClaudeUsagePause,
markRunPausedForClaudeUsage,
pauseRoomForClaudeUsage,
} from './claude-usage-pause.js';
import { getPairedTaskById } from './db.js'; import { getPairedTaskById } from './db.js';
import { completePairedTaskWithReason } from './paired-task-status.js'; import { completePairedTaskWithReason } from './paired-task-status.js';
import { prepareMessageAgentExecutionTarget } from './message-agent-executor-target.js'; import { prepareMessageAgentExecutionTarget } from './message-agent-executor-target.js';
@@ -30,9 +37,6 @@ import {
// ── Main executor ───────────────────────────────────────────────── // ── Main executor ─────────────────────────────────────────────────
/** completion_reason of a paired task closed because Claude ran out of usage. */
export const CLAUDE_USAGE_LIMIT_COMPLETION_REASON = 'claude_usage_limit';
/** completion_reason of a paired task closed because Claude's safeguards refused it. */ /** completion_reason of a paired task closed because Claude's safeguards refused it. */
export const CLAUDE_SAFETY_REFUSAL_COMPLETION_REASON = 'claude_safety_refusal'; export const CLAUDE_SAFETY_REFUSAL_COMPLETION_REASON = 'claude_safety_refusal';
@@ -74,54 +78,133 @@ interface ClaudeUsageLimitFailure {
errorText: string | null; errorText: string | null;
} }
type PauseLifecycle = Pick<
PairedExecutionLifecycle,
'updateSummary' | 'releaseForClaudeUsagePause'
>;
/** /**
* When a Claude run failed because the account is out of usage, or because * Claude is out of usage: pause the room instead of failing the turn. The
* the model's safeguards refused the request, close the paired task (retrying * paired task is left untouched and the run is marked paused so the queue
* cannot help; the next user message starts a new task) and tell the room. * keeps its messages; the scheduler re-queues the room after the reset and the
* The refusal is never handed to the reviewer/arbiter as if it were an answer. * same turn runs again.
* Returns false for any other failure. */
async function pauseRunForClaudeUsage(args: {
chatJid: string;
runId: string;
resume: ClaudeUsageResume;
summary: string;
pairedExecutionLifecycle: PauseLifecycle;
onOutput?: (output: AgentOutput) => Promise<void>;
}): Promise<void> {
const pause = pauseRoomForClaudeUsage({
chatJid: args.chatJid,
resumeAt: args.resume.resumeAt,
resetKnown: args.resume.resetKnown,
});
markRunPausedForClaudeUsage(args.runId);
args.pairedExecutionLifecycle.updateSummary({ outputText: args.summary });
args.pairedExecutionLifecycle.releaseForClaudeUsagePause();
const notice = formatClaudeUsagePauseMessage({
resumeAt: pause.until,
resetKnown: pause.resetKnown,
});
await args.onOutput?.({
status: 'success',
result: null,
output: { visibility: 'public', text: notice },
phase: 'final',
});
}
/**
* The Claude limit is account-wide: while it lasts, a Claude turn in a room
* that is not paused yet pauses it instead of starting a run that would fail.
* Returns false when no limit is in effect.
*/
async function pauseWhileClaudeUsageLimited(args: {
chatJid: string;
runId: string;
activeRole: PairedRoomRole;
pairedExecutionLifecycle: PauseLifecycle &
Pick<PairedExecutionLifecycle, 'asyncFinalize'>;
onOutput?: (output: AgentOutput) => Promise<void>;
log: { warn: (obj: Record<string, unknown>, msg: string) => void };
}): Promise<boolean> {
const pause = getActiveClaudeUsagePause();
if (!pause) return false;
args.log.warn(
{ until: pause.until.toISOString(), role: args.activeRole },
'Claude usage limit still in effect; pausing the room without starting a run',
);
try {
await pauseRunForClaudeUsage({
chatJid: args.chatJid,
runId: args.runId,
resume: { resumeAt: pause.until, resetKnown: pause.resetKnown },
summary: 'Claude usage limit still in effect',
pairedExecutionLifecycle: args.pairedExecutionLifecycle,
onOutput: args.onOutput,
});
} finally {
await args.pairedExecutionLifecycle.asyncFinalize();
}
return true;
}
/**
* When a Claude run failed because the account is out of usage, pause the
* room until the limit resets. When the model's safeguards refused the
* request, close the paired task (retrying cannot help; the next user message
* starts a new task) and tell the room; the refusal is never handed to the
* reviewer/arbiter as if it were an answer. Returns false for any other
* failure.
*/ */
async function reportClaudeUsageLimitToRoom( async function reportClaudeUsageLimitToRoom(
args: ClaudeUsageLimitFailure & { args: ClaudeUsageLimitFailure & {
isClaudeCodeAgent: boolean; isClaudeCodeAgent: boolean;
activeRole: PairedRoomRole; activeRole: PairedRoomRole;
chatJid: string;
runId: string;
pairedTaskId?: string; pairedTaskId?: string;
pairedExecutionLifecycle: Pick< pairedExecutionLifecycle: Pick<
PairedExecutionLifecycle, PairedExecutionLifecycle,
'updateSummary' | 'completeImmediately' 'updateSummary' | 'completeImmediately' | 'releaseForClaudeUsagePause'
>; >;
onOutput?: (output: AgentOutput) => Promise<void>; onOutput?: (output: AgentOutput) => Promise<void>;
log: { warn: (obj: Record<string, unknown>, msg: string) => void }; log: { warn: (obj: Record<string, unknown>, msg: string) => void };
}, },
): Promise<boolean> { ): Promise<boolean> {
if (!args.isClaudeCodeAgent) return false; if (!args.isClaudeCodeAgent) return false;
const isSafetyRefusal = args.reason === 'safety-refusal'; if (isClaudeUsageLimitReason(args.reason)) {
if (!isSafetyRefusal && !isClaudeUsageLimitReason(args.reason)) { args.log.warn(
return false; { reason: args.reason, role: args.activeRole, errorText: args.errorText },
'Claude usage limit reached; pausing the room until it resets',
);
await pauseRunForClaudeUsage({
chatJid: args.chatJid,
runId: args.runId,
resume: resolveClaudeUsageResume(args.errorText),
summary: args.errorText ?? 'Claude usage limit reached',
pairedExecutionLifecycle: args.pairedExecutionLifecycle,
onOutput: args.onOutput,
});
return true;
} }
const notice = isSafetyRefusal if (args.reason !== 'safety-refusal') return false;
? CLAUDE_SAFETY_REFUSAL_NOTICE
: buildClaudeUsageLimitMessage(args.errorText);
const task = args.pairedTaskId const task = args.pairedTaskId
? getPairedTaskById(args.pairedTaskId) ? getPairedTaskById(args.pairedTaskId)
: undefined; : undefined;
if (task) { if (task) {
completePairedTaskWithReason( completePairedTaskWithReason(task, CLAUDE_SAFETY_REFUSAL_COMPLETION_REASON);
task,
isSafetyRefusal
? CLAUDE_SAFETY_REFUSAL_COMPLETION_REASON
: CLAUDE_USAGE_LIMIT_COMPLETION_REASON,
);
} }
args.log.warn( args.log.warn(
{ reason: args.reason, role: args.activeRole, errorText: args.errorText }, { reason: args.reason, role: args.activeRole, errorText: args.errorText },
isSafetyRefusal 'Claude safeguards refused the request; closed the task and told the room',
? 'Claude safeguards refused the request; closed the task and told the room'
: 'Claude usage limit reached; told the room when it resets',
); );
await deliverNoticeAndFailExecution({ await deliverNoticeAndFailExecution({
notice, notice: CLAUDE_SAFETY_REFUSAL_NOTICE,
summary: isSafetyRefusal ? notice : (args.errorText ?? notice), summary: CLAUDE_SAFETY_REFUSAL_NOTICE,
pairedExecutionLifecycle: args.pairedExecutionLifecycle, pairedExecutionLifecycle: args.pairedExecutionLifecycle,
onOutput: args.onOutput, onOutput: args.onOutput,
}); });
@@ -286,6 +369,8 @@ export async function runAgentForGroup(
...limit, ...limit,
isClaudeCodeAgent, isClaudeCodeAgent,
activeRole, activeRole,
chatJid,
runId,
pairedTaskId: pairedExecutionContext?.task.id, pairedTaskId: pairedExecutionContext?.task.id,
pairedExecutionLifecycle, pairedExecutionLifecycle,
onOutput, onOutput,
@@ -342,6 +427,20 @@ export async function runAgentForGroup(
} }
} }
if (
isClaudeCodeAgent &&
(await pauseWhileClaudeUsageLimited({
chatJid,
runId,
activeRole,
pairedExecutionLifecycle,
onOutput,
log,
}))
) {
return 'success';
}
const runAttempt = (currentProvider: 'claude' | 'codex') => const runAttempt = (currentProvider: 'claude' | 'codex') =>
runMessageAgentAttempt({ runMessageAgentAttempt({
provider: currentProvider, provider: currentProvider,

View File

@@ -36,7 +36,12 @@ import {
RELOGIN_REQUIRED_MESSAGE, RELOGIN_REQUIRED_MESSAGE,
} from './token-refresh.js'; } from './token-refresh.js';
import { findChannel, formatMessages } from './router.js'; import { findChannel, formatMessages } from './router.js';
import { stopRoomWorkIfRequested, type RoomStopResult } from './room-stop.js'; import {
findStopRequest,
stopRoomWorkIfRequested,
type RoomStopResult,
} from './room-stop.js';
import { isRoomPausedForClaudeUsage } from './claude-usage-pause.js';
import { reopenReviewSkippedTaskForReview } from './review-request-gate.js'; import { reopenReviewSkippedTaskForReview } from './review-request-gate.js';
import { createScopedLogger, logger } from './logger.js'; import { createScopedLogger, logger } from './logger.js';
import type { GroupQueue, GroupRunContext } from './group-queue.js'; import type { GroupQueue, GroupRunContext } from './group-queue.js';
@@ -269,6 +274,18 @@ async function processMissedMessages(
args.failureFinalText, args.failureFinalText,
); );
// A stop request still goes through, so the user can drop paused work.
if (
isRoomPausedForClaudeUsage(runtime.chatJid) &&
!findStopRequest(missedMessages)
) {
runtime.log.info(
{ pendingMessages: missedMessages.length },
'Room is paused by the Claude usage limit; leaving its work for the resume',
);
return true;
}
if (missedMessages.length === 0) { if (missedMessages.length === 0) {
return handleNoMissedMessages(args, runtime, rawMissedMessages); return handleNoMissedMessages(args, runtime, rawMissedMessages);
} }

View File

@@ -1,6 +1,7 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'; import { beforeEach, describe, expect, it, vi } from 'vitest';
import { _initTestDatabase, createPairedTask } from './db.js'; import { _initTestDatabase, createPairedTask } from './db.js';
import { markRunPausedForClaudeUsage } from './claude-usage-pause.js';
import { runQueuedGroupTurn } from './message-runtime-queue.js'; import { runQueuedGroupTurn } from './message-runtime-queue.js';
import { resetPairedFollowUpScheduleState } from './paired-follow-up-scheduler.js'; import { resetPairedFollowUpScheduleState } from './paired-follow-up-scheduler.js';
import type { import type {
@@ -154,6 +155,39 @@ describe('message-runtime queued cursor handling', () => {
); );
}); });
it('keeps the queued human cursor when the Claude usage limit paused the turn', async () => {
const task = makeTask();
createPairedTask(task);
const executeTurn: RunQueuedGroupTurnArgs['executeTurn'] = vi.fn(
async () => {
// The executor posted the pause notice and marked the run paused.
markRunPausedForClaudeUsage('run-owner-queued-cursor');
return {
outputStatus: 'success' as const,
deliverySucceeded: true,
visiblePhase: 'final',
};
},
);
const lastAgentTimestamps: Record<string, string> = {};
const saveState = vi.fn(() => undefined);
const outcome = await runQueuedGroupTurn(
makeQueuedTurnArgs({
task,
executeTurn,
lastAgentTimestamps,
saveState,
}),
);
// Handled (no queue retry backoff), but the message stays unconsumed so
// the same request runs again when the room resumes.
expect(outcome).toBe(true);
expect(lastAgentTimestamps).toEqual({});
expect(saveState).not.toHaveBeenCalled();
});
it('advances the queued human cursor when a silent owner turn succeeds', async () => { it('advances the queued human cursor when a silent owner turn succeeds', async () => {
const task = makeTask({ const task = makeTask({
owner_service_id: 'codex', owner_service_id: 'codex',

View File

@@ -1,4 +1,5 @@
import { getPairedTurnOutputs } from './db.js'; import { getPairedTurnOutputs } from './db.js';
import { consumeRunPausedForClaudeUsage } from './claude-usage-pause.js';
import { logger } from './logger.js'; import { logger } from './logger.js';
import { import {
buildArbiterPromptForTask, buildArbiterPromptForTask,
@@ -442,6 +443,9 @@ export async function runQueuedGroupTurn(args: {
pairedTurnIdentity, pairedTurnIdentity,
}); });
// Paused by the Claude usage limit: keep the messages for the resume.
if (consumeRunPausedForClaudeUsage(runId)) return true;
if ( if (
!finalizeQueuedRunCursor({ !finalizeQueuedRunCursor({
outputStatus, outputStatus,

View File

@@ -104,6 +104,8 @@ vi.mock('./db.js', () => {
getLastHumanMessageContent: vi.fn(() => null), getLastHumanMessageContent: vi.fn(() => null),
getMessagesSince, getMessagesSince,
getNewMessages, getNewMessages,
getRouterState: vi.fn(() => undefined),
setRouterState: vi.fn(),
getLatestMessageSeqAtOrBefore: vi.fn(() => 0), getLatestMessageSeqAtOrBefore: vi.fn(() => 0),
getMessagesSinceSeq: vi.fn( getMessagesSinceSeq: vi.fn(
( (
@@ -375,6 +377,68 @@ describe('createMessageRuntime', () => {
expect(saveState).toHaveBeenCalled(); expect(saveState).toHaveBeenCalled();
}); });
it('leaves the messages of a room paused by the Claude usage limit for the resume', async () => {
const chatJid = 'group@test';
const group = makeGroup('claude-code');
const channel = makeChannel(chatJid);
const saveState = vi.fn();
const lastAgentTimestamps: Record<string, string> = {};
vi.mocked(db.getRouterState).mockImplementation((key) =>
key === 'claude_usage_pause'
? JSON.stringify({
until: new Date(Date.now() + 60 * 60_000).toISOString(),
resetKnown: true,
rooms: { [chatJid]: new Date().toISOString() },
})
: undefined,
);
vi.mocked(db.getMessagesSince).mockReturnValue([
{
id: 'msg-paused-1',
chat_jid: chatJid,
sender: 'user@test',
sender_name: 'User',
content: '이것도 해줘',
timestamp: '2026-10-13T03:00:00.000Z',
is_bot_message: false,
},
]);
const runtime = createMessageRuntime({
assistantName: 'Andy',
idleTimeout: 1_000,
pollInterval: 1_000,
timezone: 'UTC',
channels: [channel],
queue: {
enqueueMessageCheck: vi.fn(),
registerProcess: vi.fn(),
closeStdin: vi.fn(),
notifyIdle: vi.fn(),
} as any,
getRoomBindings: () => ({ [chatJid]: group }),
getSessions: () => ({}),
getLastTimestamp: () => '',
setLastTimestamp: vi.fn(),
getLastAgentTimestamps: () => lastAgentTimestamps,
saveState,
persistSession: vi.fn(),
clearSession: vi.fn(),
});
const result = await runtime.processGroupMessages(chatJid, {
runId: 'run-claude-usage-paused-room',
reason: 'messages',
});
expect(result).toBe(true);
expect(agentRunner.runAgentProcess).not.toHaveBeenCalled();
expect(channel.sendMessage).not.toHaveBeenCalled();
expect(lastAgentTimestamps).toEqual({});
expect(saveState).not.toHaveBeenCalled();
});
it('keeps mentionless substantive bot messages in paired rooms', async () => { it('keeps mentionless substantive bot messages in paired rooms', async () => {
const chatJid = 'group@test'; const chatJid = 'group@test';
const group = makeGroup('codex'); const group = makeGroup('codex');

View File

@@ -17,6 +17,10 @@ import {
storeMessage, storeMessage,
} from './db.js'; } from './db.js';
import { requireDatabase } from './db/runtime-database.js'; import { requireDatabase } from './db/runtime-database.js';
import {
isRoomPausedForClaudeUsage,
pauseRoomForClaudeUsage,
} from './claude-usage-pause.js';
import { processLoopGroupMessages } from './message-runtime-dispatch.js'; import { processLoopGroupMessages } from './message-runtime-dispatch.js';
import { advanceLastAgentCursor } from './message-runtime-rules.js'; import { advanceLastAgentCursor } from './message-runtime-rules.js';
import { hasReviewerLease } from './service-routing.js'; import { hasReviewerLease } from './service-routing.js';
@@ -224,6 +228,31 @@ describe('natural-language stop in the message loop', () => {
expect(args.enqueueMessageCheck).not.toHaveBeenCalled(); expect(args.enqueueMessageCheck).not.toHaveBeenCalled();
}); });
it('drops a room paused by the Claude usage limit from the resume list', async () => {
const request = postHumanMessage('배포 스크립트 고쳐줘');
const stop = postHumanMessage('중단');
pauseRoomForClaudeUsage({
chatJid,
resumeAt: new Date(Date.now() + 60 * 60_000),
resetKnown: true,
});
const controls = makeControls(false);
const lastAgentTimestamps: Record<string, string> = {
[chatJid]: String(request.seq! - 1),
};
await processLoopGroupMessages(
makeArgs({
groupMessages: [request, stop],
lastAgentTimestamps,
controls,
}),
);
expect(controls.sendReply).toHaveBeenCalledWith(chatJid, USER_STOP_REPLY);
expect(isRoomPausedForClaudeUsage(chatJid)).toBe(false);
});
it('still processes a new instruction sent after the stop', async () => { it('still processes a new instruction sent after the stop', async () => {
const stop = postHumanMessage('중단'); const stop = postHumanMessage('중단');
const next = postHumanMessage('대신 테스트만 돌려줘'); const next = postHumanMessage('대신 테스트만 돌려줘');

View File

@@ -5,6 +5,7 @@ import {
getPairedTurnsForTask, getPairedTurnsForTask,
releasePairedTaskExecutionLeasesForTask, releasePairedTaskExecutionLeasesForTask,
} from './db.js'; } from './db.js';
import { releaseRoomFromClaudeUsagePause } from './claude-usage-pause.js';
import { logger } from './logger.js'; import { logger } from './logger.js';
import { import {
advanceLastAgentCursor, advanceLastAgentCursor,
@@ -185,6 +186,7 @@ export async function stopRoomWorkIfRequested(args: {
saveState: args.saveState, saveState: args.saveState,
}); });
const closedTask = openTask ? closePairedTaskForStop(openTask) : null; const closedTask = openTask ? closePairedTaskForStop(openTask) : null;
const releasedUsagePause = releaseRoomFromClaudeUsagePause(chatJid);
const killedProcess = controls.stopRunningAgent(chatJid); const killedProcess = controls.stopRunningAgent(chatJid);
logger.info( logger.info(
@@ -198,6 +200,7 @@ export async function stopRoomWorkIfRequested(args: {
hadPendingMessages: hasPendingMessages, hadPendingMessages: hasPendingMessages,
pairedTaskId: openTask?.id ?? null, pairedTaskId: openTask?.id ?? null,
pairedTaskStatus: openTask?.status ?? null, pairedTaskStatus: openTask?.status ?? null,
releasedUsagePause,
...(closedTask ?? {}), ...(closedTask ?? {}),
}, },
'Stopped room work on user request', 'Stopped room work on user request',

View File

@@ -125,7 +125,14 @@ import {
createPairedTask, createPairedTask,
createTask, createTask,
getTaskById, getTaskById,
setRouterState,
} from './db.js'; } from './db.js';
import {
CLAUDE_USAGE_PAUSE_KEY,
CLAUDE_USAGE_RESUME_NOTICE,
isRoomPausedForClaudeUsage,
pauseRoomForClaudeUsage,
} from './claude-usage-pause.js';
import { TIMEZONE } from './config.js'; import { TIMEZONE } from './config.js';
import * as serviceRouting from './service-routing.js'; import * as serviceRouting from './service-routing.js';
import * as tokenRefresh from './token-refresh.js'; import * as tokenRefresh from './token-refresh.js';
@@ -321,6 +328,59 @@ describe('task scheduler', () => {
expect(enqueueMessageCheck).toHaveBeenCalledTimes(1); expect(enqueueMessageCheck).toHaveBeenCalledTimes(1);
}); });
it('resumes rooms paused by the Claude usage limit once it has reset', async () => {
pauseRoomForClaudeUsage({
chatJid: 'paused@g.us',
resumeAt: new Date(Date.now() + 60 * 60_000),
resetKnown: true,
});
const enqueueMessageCheck = vi.fn();
const sendMessage = vi.fn(async () => {});
const deps = {
serviceAgentType: 'claude-code' as const,
roomBindings: () => ({
'paused@g.us': {
name: 'Paused',
folder: 'paused-group',
trigger: '@Andy',
added_at: '2026-10-13T00:00:00.000Z',
},
}),
getSessions: () => ({}),
queue: { enqueueTask: vi.fn(), enqueueMessageCheck } as any,
onProcess: () => {},
sendMessage,
};
await runSchedulerTickOnce(deps);
expect(enqueueMessageCheck).not.toHaveBeenCalled();
expect(sendMessage).not.toHaveBeenCalled();
// The limit has reset (the stored reset time is now in the past).
setRouterState(
CLAUDE_USAGE_PAUSE_KEY,
JSON.stringify({
until: new Date(Date.now() - 2 * 60_000).toISOString(),
resetKnown: true,
rooms: { 'paused@g.us': new Date().toISOString() },
}),
);
await runSchedulerTickOnce(deps);
expect(sendMessage).toHaveBeenCalledWith(
'paused@g.us',
CLAUDE_USAGE_RESUME_NOTICE,
);
expect(enqueueMessageCheck).toHaveBeenCalledWith(
'paused@g.us',
expect.stringContaining('paused-group'),
);
expect(isRoomPausedForClaudeUsage('paused@g.us')).toBe(false);
await runSchedulerTickOnce(deps);
expect(enqueueMessageCheck).toHaveBeenCalledTimes(1);
});
it('can execute one scheduler tick without starting the timer loop', async () => { it('can execute one scheduler tick without starting the timer loop', async () => {
const dueAt = new Date(Date.now() - 60_000).toISOString(); const dueAt = new Date(Date.now() - 60_000).toISOString();
createTask({ createTask({

View File

@@ -34,6 +34,10 @@ import { getTaskQueueJid, isGitHubCiTask } from './task-watch-status.js';
import { normalizeAgentOutputPhase, ScheduledTask } from './types.js'; import { normalizeAgentOutputPhase, ScheduledTask } from './types.js';
import { resolveGroupIpcPath } from './group-folder.js'; import { resolveGroupIpcPath } from './group-folder.js';
import { schedulePairedFollowUpOnce } from './paired-follow-up-scheduler.js'; import { schedulePairedFollowUpOnce } from './paired-follow-up-scheduler.js';
import {
CLAUDE_USAGE_RESUME_NOTICE,
takeRoomsReadyToResume,
} from './claude-usage-pause.js';
import { import {
hasTaskExceededMaxDuration, hasTaskExceededMaxDuration,
resolveTaskExecutionContext, resolveTaskExecutionContext,
@@ -429,6 +433,7 @@ export async function runSchedulerTickOnce(
): Promise<void> { ): Promise<void> {
// Unified service: process all agent types, not just the service default. // Unified service: process all agent types, not just the service default.
const nowMs = Date.now(); const nowMs = Date.now();
await resumeClaudeUsagePausedRooms(deps, nowMs);
reconcileOrphanedPairedReviewReadyTasks(deps); reconcileOrphanedPairedReviewReadyTasks(deps);
const activeTasks = getAllTasks().filter((task) => task.status === 'active'); const activeTasks = getAllTasks().filter((task) => task.status === 'active');
@@ -474,6 +479,39 @@ export async function runSchedulerTickOnce(
} }
} }
/** Re-queue rooms paused by the Claude usage limit once it has reset. */
async function resumeClaudeUsagePausedRooms(
deps: SchedulerDependencies,
nowMs: number,
): Promise<void> {
const chatJids = takeRoomsReadyToResume(nowMs);
if (chatJids.length === 0) return;
const bindings = deps.roomBindings();
for (const chatJid of chatJids) {
const binding = bindings[chatJid];
if (!binding) {
logger.warn(
{ chatJid },
'Skipped resuming a Claude-usage-paused room without a room binding',
);
continue;
}
try {
await deps.sendMessage(chatJid, CLAUDE_USAGE_RESUME_NOTICE);
} catch (err) {
logger.warn({ chatJid, err }, 'Failed to send the Claude resume notice');
}
deps.queue.enqueueMessageCheck(
chatJid,
resolveGroupIpcPath(binding.folder),
);
logger.info(
{ chatJid, groupFolder: binding.folder },
'Resumed room work paused by the Claude usage limit',
);
}
}
function reconcileOrphanedPairedReviewReadyTasks( function reconcileOrphanedPairedReviewReadyTasks(
deps: SchedulerDependencies, deps: SchedulerDependencies,
): void { ): void {