fix: rewrite English final answers in Korean before delivery

A sketchup-app turn spent on English login pages (Trimble/Edge) ended
with an English NEEDS_CONTEXT answer. The Korean rule was only a prompt
line, the answer went straight to the user (no reviewer gate), and it was
carried forward into the next task, so the next turn answered in English
too.

- Claude runner: when a final answer is mostly English (Hangul < 10% of
  Latin letters, >= 150 letters, code/links/boilerplate ignored), ask the
  same session once to rewrite it in Korean and emit only the rewrite.
  Falls back to the original on error or early stream end; disabled when
  AGENT_LANGUAGE is set to a non-Korean language.
- Let handleResultMessage settle the buffered final text instead of
  pre-flushing it as progress, so the English draft never surfaces.
- Platform prompt: explicit "Reply language" section; owner rule no longer
  says "unless the user wrote in another language".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codex
2026-10-24 02:07:17 +09:00
parent e3308eb119
commit 98011e282b
5 changed files with 336 additions and 3 deletions

View File

@@ -5,6 +5,14 @@ Use it to acknowledge a request before starting longer work.
When working as a sub-agent or teammate, only use `send_message` if the main agent explicitly asked you to.
## Reply language
Write everything the user sees in Korean (존댓말): the final answer, progress text, and `send_message` messages. Use another language only when the user explicitly asks for it.
This holds even when the work itself is in English: tool output, web pages, screenshots, English UI, English credentials in the user's message, or a previous answer that was carried forward in English. None of these is a request for an English reply. Code, commands, file paths, identifiers, and the exact text of on-screen buttons may stay as they are.
A final answer that comes out mostly in English is sent back to you once to be rewritten in Korean, which costs an extra turn.
## Background work never notifies you — finish within the turn
Nothing in this environment re-invokes you when a background process, a background shell command, or a `run_in_background` job finishes. You will NOT be automatically notified when a build, compile, download, deploy, or any background task completes. The built-in tool descriptions that say "you'll be notified when it finishes" do NOT apply here.

View File

@@ -80,7 +80,7 @@ Do **not** include:
- A blow-by-blow of each round.
- Apologies for earlier owner mistakes that the user never saw.
The default reply language is the user's language (Korean for this deployment) unless the user wrote in another language.
Write the reply in Korean unless the user explicitly asks for another language. See "Reply language" in the platform rules.
## Rules

View File

@@ -24,6 +24,11 @@ import {
type RunnerCompaction,
writeOutput,
} from './output-protocol.js';
import {
KOREAN_REWRITE_NUDGE,
isKoreanReplyTarget,
needsKoreanRewrite,
} from './reply-language-guard.js';
import { buildClaudeReadonlySandboxSettings } from './reviewer-runtime.js';
import {
createPreCompactHook,
@@ -84,6 +89,14 @@ class ClaudeQueryRunner {
private terminalResultObserved = false;
private pendingProgressText: string | null = null;
private compaction: RunnerCompaction | undefined;
private readonly languageGuardEnabled = isKoreanReplyTarget(
process.env.AGENT_LANGUAGE,
);
private languageRewrite: 'none' | 'pending' | 'done' = 'none';
private languageRewriteSource: string | null = null;
// Set when the rewrite was requested from the end_turn assistant shortcut:
// that turn's own result event still follows and must not be emitted.
private staleResultExpected = false;
constructor(private readonly args: RunClaudeQueryArgs) {
this.stream = new MessageStream((text) =>
@@ -160,6 +173,7 @@ class ClaudeQueryRunner {
private handleCloseSentinel(): void {
this.args.log('Close sentinel detected during query, ending stream');
this.flushLanguageRewriteFallback();
if (this.pendingProgressText && !this.terminalResultObserved) {
this.args.log(
`Flushing pending text before close (${this.pendingProgressText.length} chars)`,
@@ -383,7 +397,10 @@ class ClaudeQueryRunner {
private handleMessage(message: unknown): boolean {
const type = (message as { type?: string }).type;
if (type !== 'assistant' && this.pendingProgressText) {
// A result event settles the buffered text itself (handleResultMessage):
// flushing it here first would surface the final answer as progress, even
// an English one that is about to be rewritten.
if (type !== 'assistant' && type !== 'result' && this.pendingProgressText) {
this.flushPendingProgressAsIntermediate();
}
if (type === 'system') this.handleSystemMessage(message);
@@ -495,6 +512,14 @@ class ClaudeQueryRunner {
let textResult = resultMessage.result || null;
const isError = resultMessage.subtype?.startsWith('error');
if (this.languageRewrite === 'pending' && this.staleResultExpected) {
this.staleResultExpected = false;
// Result event of the English answer we already asked to rewrite.
this.pendingProgressText = null;
this.args.log('Ignoring result of the answer being rewritten in Korean');
return false;
}
if (this.pendingProgressText && textResult === this.pendingProgressText) {
this.args.log('Discarding pending progress (matches result)');
this.pendingProgressText = null;
@@ -518,9 +543,14 @@ class ClaudeQueryRunner {
this.args.log(
`Result #${this.resultCount}: subtype=${resultMessage.subtype}${textResult ? ` text=${textResult.slice(0, 200)}` : ''}`,
);
if (isError) {
if (isError && this.languageRewrite === 'pending') {
this.args.log('Korean rewrite failed, emitting the original answer');
this.writeSuccessResult(this.languageRewriteSource);
this.languageRewrite = 'done';
} else if (isError) {
this.writeErrorResult(resultMessage, textResult);
} else {
if (this.requestKoreanRewrite(textResult, false)) return false;
this.writeSuccessResult(textResult);
}
this.terminalResultObserved = true;
@@ -578,6 +608,7 @@ class ClaudeQueryRunner {
}
private handleAssistantMessage(message: unknown): boolean {
if (this.languageRewrite === 'pending') this.staleResultExpected = false;
this.trackedAgentTasks.rememberAssistantMessage(message);
this.toolUsage.recordAssistantMessage(message);
const stopReason = (message as { stop_reason?: string }).stop_reason;
@@ -588,6 +619,7 @@ class ClaudeQueryRunner {
);
}
if (stopReason === 'end_turn' && textResult) {
if (this.requestKoreanRewrite(textResult, true)) return false;
this.resultCount++;
this.args.log(
`Terminal assistant turn observed without result event (${textResult.length} chars), ending query stream`,
@@ -614,6 +646,42 @@ class ClaudeQueryRunner {
return false;
}
/**
* When a final answer came out in English, ask the same session once to
* rewrite it in Korean and keep the stream open for the rewrite. Returns true
* when the rewrite was requested (the caller must not emit `text`).
*/
private requestKoreanRewrite(
text: string | null,
resultStillPending: boolean,
): boolean {
if (!this.languageGuardEnabled || this.languageRewrite !== 'none') {
if (this.languageRewrite === 'pending') this.languageRewrite = 'done';
return false;
}
if (this.closedDuringQuery || !needsKoreanRewrite(text)) return false;
this.languageRewrite = 'pending';
this.languageRewriteSource = text;
this.staleResultExpected = resultStillPending;
this.args.log(
`Final answer looks English (${text!.length} chars), asking for a Korean rewrite`,
);
this.stream.push(KOREAN_REWRITE_NUDGE);
return true;
}
/** Emit the original answer if the stream ends before the rewrite arrives. */
private flushLanguageRewriteFallback(): void {
if (this.languageRewrite !== 'pending' || this.terminalResultObserved) {
return;
}
this.args.log('Korean rewrite not received, emitting the original answer');
this.writeSuccessResult(this.languageRewriteSource);
this.languageRewrite = 'done';
this.terminalResultObserved = true;
this.resultCount++;
}
private flushPendingProgressAsIntermediate(): void {
if (!this.pendingProgressText) return;
writeOutput({
@@ -626,6 +694,7 @@ class ClaudeQueryRunner {
}
private flushRemainingPendingText(): void {
this.flushLanguageRewriteFallback();
if (!this.pendingProgressText || this.terminalResultObserved) return;
this.args.log(
`Flushing remaining pending progress text as final output (${this.pendingProgressText.length} chars)`,

View File

@@ -0,0 +1,61 @@
// Guard against final answers that drift into English.
//
// User-facing replies in this deployment must be Korean. The prompt says so, but
// a prompt rule cannot reliably hold during long sessions full of English tool
// output, web pages, and screenshots: the model sometimes writes its final answer
// in English, and that answer is then carried forward into the next task's
// prompt, so the following turns stay English too (incident 2026-10-24, room
// sketchup-app). So the runner also checks the final text: when it is clearly
// English, it asks the same session once to rewrite it in Korean and emits the
// rewrite instead.
// Below this many Latin letters the text is too short to judge.
const MIN_LATIN_LETTERS = 150;
// Clearly-English answers measured at <= 8% Hangul-to-Latin; normal Korean
// answers full of identifiers stay above ~17%.
const MAX_HANGUL_TO_LATIN_RATIO = 0.1;
const HOST_ERROR_PREFIXES = ['API Error', 'Prompt is too long'];
export const KOREAN_REWRITE_NUDGE =
'[EJClaw 자동 언어 점검] 방금 최종 답변이 영어로 작성됐습니다. 이 서비스의 사용자 응답 언어는 한국어입니다. ' +
'사용자가 이 대화에서 영어 답변을 명시적으로 요청한 경우가 아니라면, 방금 답변을 한국어 존댓말로 다시 써서 그 답변만 출력하세요. ' +
'내용은 바꾸거나 줄이지 말고, 첫 줄의 상태 키워드(TASK_DONE, STEP_DONE, NEEDS_CONTEXT, REVISE 등), 코드, 명령어, 파일 경로, 링크, MEDIA: 줄, 화면에 보이는 영문 버튼·메뉴 이름은 그대로 두세요. ' +
'도구는 쓰지 마세요. 사용자가 영어를 명시적으로 요청했다면 방금 답변을 그대로 다시 출력하세요.';
function stripNonProse(text: string): string {
return (
text
.replace(/```[\s\S]*?```/g, ' ')
.replace(/`[^`\n]*`/g, ' ')
.replace(/https?:\/\/\S+/g, ' ')
.replace(/^MEDIA:.*$/gm, ' ')
// Fixed Korean boilerplate an English answer may still carry.
.replace(/^리뷰 불필요:.*$/gm, ' ')
.replace(/사용자 액션 아이템(?:: 없음)?/g, ' ')
);
}
/** True when the guard should run for this deployment's configured language. */
export function isKoreanReplyTarget(
agentLanguage: string | undefined,
): boolean {
const target = agentLanguage?.trim();
if (!target) return true;
return /^(ko|ko-kr|korean|한국어)$/i.test(target);
}
/** True when `text` is a final answer written in English instead of Korean. */
export function needsKoreanRewrite(text: string | null | undefined): boolean {
if (!text) return false;
const trimmed = text.trim();
if (trimmed.startsWith('{')) return false;
if (HOST_ERROR_PREFIXES.some((prefix) => trimmed.startsWith(prefix))) {
return false;
}
const prose = stripNonProse(trimmed);
const latin = prose.match(/[A-Za-z]/g)?.length ?? 0;
if (latin < MIN_LATIN_LETTERS) return false;
const hangul = prose.match(/[가-힣]/g)?.length ?? 0;
return hangul < latin * MAX_HANGUL_TO_LATIN_RATIO;
}

View File

@@ -0,0 +1,195 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
const outputs: Array<{ status: string; result: string | null }> = [];
let scriptedTurns: Array<Array<Record<string, unknown>>> = [];
const promptsSeen: string[] = [];
vi.mock('@anthropic-ai/claude-agent-sdk', () => ({
query: ({ prompt }: { prompt: AsyncIterable<unknown> }) =>
(async function* () {
yield { type: 'system', subtype: 'init', session_id: 'sess-1' };
for await (const message of prompt) {
const content = (message as { message: { content: unknown } }).message
.content;
promptsSeen.push(
typeof content === 'string' ? content : JSON.stringify(content),
);
const turn = scriptedTurns.shift();
if (!turn) return; // CLI exits without answering
for (const event of turn) yield event;
}
})(),
}));
vi.mock('../src/claude-cli.js', () => ({
getClaudeCliPath: () => '/fake/claude',
}));
vi.mock('../src/output-protocol.js', async (importOriginal) => {
const actual =
await importOriginal<typeof import('../src/output-protocol.js')>();
return {
...actual,
writeOutput: (output: { status: string; result: string | null }) => {
outputs.push({ status: output.status, result: output.result });
},
};
});
import { runClaudeQuery } from '../src/claude-query-runner.js';
import {
KOREAN_REWRITE_NUDGE,
isKoreanReplyTarget,
needsKoreanRewrite,
} from '../src/reply-language-guard.js';
const ENGLISH_FINAL =
'NEEDS_CONTEXT\n\nI accepted the Terms and entered the email and password, but Trimble now wants a verification code it emailed to you, so sign-in isn\'t finished yet.\n\n- I clicked "Accept" on the Trimble Offering Terms.\n- On the Trimble ID page, I rejected the cookie banner and entered your email.\n\n사용자 액션 아이템:\n1. Please send me the verification code.';
const KOREAN_FINAL =
'NEEDS_CONTEXT\n\n약관에 동의하고 이메일과 비밀번호를 입력했지만, Trimble이 이메일로 보낸 인증 코드를 요구해서 아직 로그인이 끝나지 않았습니다.';
// Real SDK assistant messages carry stop_reason only inside `message` (null
// while streaming); `topLevelEndTurn` exercises the runner's end_turn shortcut.
function assistant(text: string, topLevelEndTurn = false) {
return {
type: 'assistant',
...(topLevelEndTurn ? { stop_reason: 'end_turn' } : {}),
message: { content: [{ type: 'text', text }], stop_reason: null },
};
}
function result(text: string, subtype = 'success') {
return { type: 'result', subtype, result: text };
}
async function run() {
return runClaudeQuery({
prompt: '사용자 요청',
sessionId: undefined,
mcpServerPath: '/fake/ipc-mcp-stdio.js',
runnerInput: { prompt: '사용자 요청', groupFolder: 'g', chatJid: 'dc:1' },
sdkEnv: {},
reviewerRuntime: false,
claudeReadonlyReviewerRuntime: false,
claudeReadonlySandboxMode: null,
abortController: new AbortController(),
paths: {
groupDir: '/tmp/ejclaw-lang-guard-test',
groupFolder: 'g',
hostTasksDir: '/tmp/ejclaw-lang-guard-test/tasks',
ipcInputCloseSentinel: '/tmp/ejclaw-lang-guard-test/_close',
ipcInputDir: '/tmp/ejclaw-lang-guard-test/input',
workDir: '',
},
log: () => {},
});
}
const visible = () => outputs.map((o) => [o.status, o.result]);
describe('needsKoreanRewrite', () => {
it('flags an English final answer', () => {
expect(needsKoreanRewrite(ENGLISH_FINAL)).toBe(true);
});
it('does not let Korean boilerplate lines hide an English answer', () => {
const english =
'TASK_DONE\n\nRelease 2.4.0 is out. The importer now keeps section planes, scene tabs and layer visibility when opening older files. Shadows render faster on machines without a graphics card, and the measure tool snaps to arc centers.\n\n사용자 액션 아이템: 없음\n리뷰 불필요: 요청받은 문구를 그대로 전달했을 뿐 변경 사항이 없습니다.';
expect(needsKoreanRewrite(english)).toBe(true);
});
it('keeps Korean answers full of identifiers, code and links', () => {
const korean =
'TASK_DONE\n\nv0.3.5 배포 완료했습니다. `src/installer-rp/pack.ts`의 `electron-builder` 설정을 고쳤습니다.\n\n```bash\nbun run build && git push origin main --follow-tags\n```\n릴리즈: https://git.tkrmagid.kr/tkrmagid/minecraft-resourcepack/releases/tag/v0.3.5\n- origin/main = 9efd4a6, CI green, assets uploaded';
expect(needsKoreanRewrite(korean)).toBe(false);
});
it('ignores short text, JSON and host error banners', () => {
expect(needsKoreanRewrite('TASK_DONE\n\nDone.')).toBe(false);
expect(needsKoreanRewrite(`{"answer": "${'x'.repeat(300)}"}`)).toBe(false);
expect(
needsKoreanRewrite(
`API Error: Opus 5.5's safeguards flagged this message. ${'word '.repeat(60)}`,
),
).toBe(false);
});
it('is only enabled for a Korean (or unset) AGENT_LANGUAGE', () => {
expect(isKoreanReplyTarget(undefined)).toBe(true);
expect(isKoreanReplyTarget('')).toBe(true);
expect(isKoreanReplyTarget('Korean')).toBe(true);
expect(isKoreanReplyTarget('ko')).toBe(true);
expect(isKoreanReplyTarget('English')).toBe(false);
});
});
describe('Claude runner Korean rewrite', () => {
beforeEach(() => {
outputs.length = 0;
promptsSeen.length = 0;
delete process.env.AGENT_LANGUAGE;
});
afterEach(() => {
scriptedTurns = [];
});
it('asks once for a Korean rewrite and emits only the rewrite', async () => {
scriptedTurns = [
[assistant(ENGLISH_FINAL), result(ENGLISH_FINAL)],
[assistant(KOREAN_FINAL), result(KOREAN_FINAL)],
];
await run();
expect(promptsSeen).toEqual(['사용자 요청', KOREAN_REWRITE_NUDGE]);
expect(visible()).toEqual([['success', KOREAN_FINAL]]);
});
it('also catches the end_turn shortcut and ignores the stale result', async () => {
scriptedTurns = [
[assistant(ENGLISH_FINAL, true), result(ENGLISH_FINAL)],
[assistant(KOREAN_FINAL, true), result(KOREAN_FINAL)],
];
await run();
expect(promptsSeen).toEqual(['사용자 요청', KOREAN_REWRITE_NUDGE]);
expect(visible()).toEqual([['success', KOREAN_FINAL]]);
});
it('does not loop when the rewrite is still English', async () => {
const stillEnglish = `${ENGLISH_FINAL} (unchanged)`;
scriptedTurns = [
[assistant(ENGLISH_FINAL), result(ENGLISH_FINAL)],
[assistant(stillEnglish), result(stillEnglish)],
];
await run();
expect(promptsSeen).toHaveLength(2);
expect(visible()).toEqual([['success', stillEnglish]]);
});
it('falls back to the original answer when the rewrite errors', async () => {
scriptedTurns = [
[assistant(ENGLISH_FINAL), result(ENGLISH_FINAL)],
[result('', 'error_during_execution')],
];
await run();
expect(visible()).toEqual([['success', ENGLISH_FINAL]]);
});
it('falls back to the original answer when the stream ends early', async () => {
scriptedTurns = [[assistant(ENGLISH_FINAL), result(ENGLISH_FINAL)]];
await run();
expect(visible()).toEqual([['success', ENGLISH_FINAL]]);
});
it('leaves Korean answers alone and emits them once', async () => {
scriptedTurns = [[assistant(KOREAN_FINAL), result(KOREAN_FINAL)]];
await run();
expect(promptsSeen).toEqual(['사용자 요청']);
expect(visible()).toEqual([['success', KOREAN_FINAL]]);
});
it('is disabled when AGENT_LANGUAGE is another language', async () => {
process.env.AGENT_LANGUAGE = 'English';
scriptedTurns = [[assistant(ENGLISH_FINAL), result(ENGLISH_FINAL)]];
await run();
expect(promptsSeen).toEqual(['사용자 요청']);
expect(visible()).toEqual([['success', ENGLISH_FINAL]]);
});
});