docs: slim the shared global memory and move details to ops notes

The global memory was ~40KB (~11k tokens) and loaded into every agent call.
Keep only rules every room needs and move host, credential, EJClaw ops and
project notes to groups/global/ops/*.md, referenced on demand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codex
2026-10-01 20:52:33 +09:00
parent 50f149dd0f
commit 2e285c5d78

View File

@@ -1,287 +1,37 @@
# Global Memory
This file is for mutable memory shared across Claude groups.
Shared, mutable memory for every room. Keep it short: only rules every room needs.
Detailed notes live in `/home/claude/EJClaw/groups/global/ops/` — read the matching file when a task touches that topic:
Use it for durable facts, preferences, and shared context that may change over time.
- `ops/host.md` — GPU (RTX 5050, `/home/claude/gpu-venv`, NVML mismatch fix), tmpfs/RAM rules and watchdogs, Windows CI VM on .9 (nested KVM, resource caps), .5 Docker host and `docker5` context
- `ops/credentials.md` — what is in `secrets.json`, git.tkrmagid.kr credential helper, Claude/Codex per-service credentials and re-login runbooks
- `ops/ejclaw-ops.md` — room mode policy, channel register/deregister, safe restarts, model upgrade checklist, current versions, recent EJClaw fixes
- `ops/projects.md` — javis_bot deployment target, washing_machine_app release process
Do not store platform-wide operating rules here. Those now live in `prompts/claude-platform.md`.
## User preferences
## Host hardware (GPU)
- Answer the user in Korean 존댓말 unless asked otherwise.
- GPU: ask before running GPU workloads, except for the `javis_bot` project (pre-approved).
This host has an NVIDIA GPU available for compute:
- GPU: NVIDIA GeForce RTX 5050 (GB207, Blackwell), 8GB VRAM, compute capability 12.0 (sm_120)
- Driver 595.x (open kernel module), CUDA runtime 13.2; `libcuda.so` present so GPU compute is usable
- Driver packages auto-update via apt. After an update the userspace libs move ahead of the still-loaded kernel module, and `nvidia-smi` fails with `Failed to initialize NVML: Driver/library version mismatch` (CUDA compute through `/home/claude/gpu-venv` keeps working meanwhile). Fix without a reboot when `modinfo -F version nvidia` already equals the new userspace version: stop the GPU holders (`systemctl --user stop wsai-voice vncserver`, `sudo systemctl stop gdm ollama`; check `sudo fuser /dev/nvidia*` is empty), `sudo modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia`, `sudo modprobe nvidia nvidia_uvm nvidia_modeset nvidia_drm`, then start the four services again. Done 2026-10-01 (595.84 → 595.91.07, ~2 min of GPU/desktop downtime; ejclaw/kaclaw unaffected). A newer kernel (7.0.0-34) is also installed and only takes effect on reboot.
- Ready-to-use PyTorch (CUDA 12.8 / Blackwell-capable) venv at `/home/claude/gpu-venv` — run with `/home/claude/gpu-venv/bin/python`. `torch.cuda.is_available()` is True.
- For CUDA C compilation, `nvcc` is not installed; install the CUDA Toolkit only if a task needs it.
## Host safety (always)
Behavioral rule: when a task could meaningfully benefit from the GPU (ML inference/training, heavy parallel compute, large media processing), first ask the user whether they want to use the GPU before doing so. Do not silently run GPU workloads. Once confirmed, use the `/home/claude/gpu-venv` Python (or install what the task needs) and verify with a real GPU run.
- `/tmp` and `/dev/shm` are RAM (tmpfs). Never put venvs, models, datasets or other large files there; use disk paths under `$HOME` (`TMPDIR=$HOME/.tmp`).
- Heavy jobs (GPU, VMs, big builds) must not run inside the bot's cgroup: launch them with `systemd-run --user --scope -p MemoryMax=... ...`.
- Never run a plain `systemctl --user restart ejclaw.service` inside a turn (it kills you). Room binding changes use SIGHUP; code/env changes use the detached `systemd-run` restart. Verify before claiming success. Details: `ops/ejclaw-ops.md`.
- 2026-06-18 exception (javis_bot project): the user pre-approved GPU use for the `javis_bot` project — do NOT ask before using the GPU on javis_bot tasks. Other projects still follow the ask-first rule above.
## Sending files
## javis_bot deployment target
Only files under `/home/claude/EJClaw/data/attachments` can be attached. Copy the file there (prefer `data/attachments/generated/`, auto-deleted after 60 min) and reference it with a `MEDIA:` line. Do not widen the allowlist.
- 2026-06-18: the real production deployment runs on the user's Windows 11 gaming PC (CPU Ryzen 9 9950X3D, GPU RTX 5070Ti — Blackwell, same sm_120 arch as this host's RTX 5050) with Docker. Use `docker-compose.yml` + `docker-compose.gpu-windows.yml` (Docker Desktop + WSL2 GPU). This Linux host is only the LAN browser host (JARVIS_ROLE=browser); test changes here as a Blackwell-GPU proxy but remember the bot/brain role actually runs on the Windows box. The `1516396486265405548` value the user pasted on 2026-06-18 was just a Discord message ID in the chat channel, not a guild/voice-channel ID.
## Credentials
## Host RAM / disk safety (tmpfs rule)
Stored at `/home/claude/.config/ejclaw/secrets.json` (read with the Read tool when needed). Never paste a secret into chat or into memory files. Each entry's `note` explains its usage rules; see `ops/credentials.md`.
`/tmp` (16G) and `/dev/shm` (16G) are RAM-backed tmpfs. Anything large written there consumes RAM, not disk.
## Rooms
- 2026-06-11 incident: MeloTTS Korean-TTS venv (~2.4GB) was installed into `/tmp/melo311` (tmpfs), eating RAM and risking OOM. Fixed by moving it to disk at `/home/claude/jarvis-tts/melo311` and updating 43 script shebangs.
- Recurrence guards (in place, defence-in-depth):
1. `~/.bashrc` exports `TMPDIR=$HOME/.tmp`, `UV_CACHE_DIR=$HOME/.cache/uv`, `PIP_CACHE_DIR=$HOME/.cache/pip` (all ext4 disk), above the interactive-shell guard so non-interactive agent shells inherit it.
2. The `ejclaw.service` user unit (`~/.config/systemd/user/ejclaw.service`) sets the same three `Environment=` vars, so every child process (agent runners, install/build steps that never source `.bashrc`) inherits disk temp at the systemd root. Applied via `daemon-reload`; takes full effect on the next `ejclaw.service` restart. Do NOT restart it from inside a live agent turn (the agent is its descendant).
3. Watchdog `/home/claude/tmpfs-ram-watchdog.sh` (user crontab `*/30`) posts once per episode to the EJClaw #상태 channel when `/tmp` or `/dev/shm` usage > 3GB or MemAvailable < 4GB, re-arms after recovery, and deletes `$HOME/.tmp` files older than 3 days. Token-free (Discord REST with the owner bot token, no agent run); log `/home/claude/tmpfs-ram-watchdog.log`, latches under `~/.tmpfs-ram-watchdog/`. Test with `DRYRUN=1 TMPFS_MAX_MB=100 /home/claude/tmpfs-ram-watchdog.sh`. This catches an explicit `/tmp/...` write that bypasses `TMPDIR`. (The original watchdog had silently disappeared; restored 2026-10-01.) Disk space is watched separately by `/home/claude/disk85-alert.sh` (cron `*/10`, @everyone alert at ≥85% root usage).
- Rule: NEVER install venvs, models, datasets, or any multi-GB artefact into `/tmp` or `/dev/shm`. Use disk paths under `$HOME` (e.g. `/home/claude/jarvis-tts`). If a tool ignores `TMPDIR`, pass an explicit disk path. EJClaw runs directly via `bun` under the user systemd unit (no Docker/compose in use here).
- Root filesystem `/` (ext4) is 506G total, ~221G free (55% used) as of 2026-10-01 (it was 156G in early June, then 353G). RAM is 30GB as seen by the guest (Proxmox shows 32GiB and counts guest page cache as "used", so its ~85% figure is misleading — judge by `free -h` → `available`). Adding RAM is not the real fix for tmpfs blowups; keeping big artefacts on disk is.
- The two TTS scratch scripts (`/home/claude/jarvis-tts/gen_melo.py`, `gen_xtts.py`) now route output to disk and hard-refuse any `/tmp` or `/dev/shm` write path via a `disk_path()` guard.
- 2026-07-26 OOM incident + `MemoryHigh` change: the `ejclaw.service` base unit set `MemoryHigh=3G` (soft cgroup cap; `MemoryMax` unset). An agent-launched GPU/diffusers job in the `anyway2` channel (`gen_banner.py`, ~2.9GB RSS on its own) ran INSIDE the bot's cgroup, pushed total past 3G, and `systemd-oomd` oom-killed `ejclaw.service` ~6× in a row (systemd auto-restarted each time → restart loop; the banner also never finished). Fix: raised the soft cap to `MemoryHigh=16G` via a documented drop-in `~/.config/systemd/user/ejclaw.service.d/10-memory-high.conf` (applied with `daemon-reload`, no restart needed for a cgroup-property change). Rationale: host has 30G, KAclaw is capped at 3G and other services ~2G, so 16G leaves ~11G+cache headroom while still capping runaways; `MemoryMax` stays infinity so a burst is throttled, not hard-killed. The base unit still reads 3G — the drop-in overrides it (`systemctl --user cat ejclaw` shows both). Cleaner long-term alternative (not done): keep the bot cap tight and launch heavy GPU jobs OUTSIDE the bot cgroup via `systemd-run --user --scope`.
## Outbound attachment allowed folders
Discord/outbound attachments are only delivered if the file lives inside an allowlisted directory; files elsewhere are rejected as `outside-allowed-dirs` and the bot now appends a visible "전송 실패" notice (it no longer drops silently).
**Standing rule (2026-06-12 user decision):** the single canonical outbound attachment folder is `/home/claude/EJClaw/data/attachments` (the EJClaw DATA_DIR `attachments` dir). It is already allow-listed by default — no `.env` registration is needed. To send ANY file, first write or copy it into that folder (a subfolder is fine, e.g. `data/attachments/generated/`), then reference that path in the `MEDIA:` directive. Do NOT widen the allowlist to other folders (the earlier idea of allow-listing `/home/claude/jarvis-tts` was reverted at the user's request, and auto-copying arbitrary out-of-allowlist files in the delivery layer was also reverted because it bypasses cross-room / sensitive-file protection).
- Concretely: when attaching generated media (TTS audio, images, etc.), `cp` it into `/home/claude/EJClaw/data/attachments/` (or a subfolder) and emit `MEDIA:/home/claude/EJClaw/data/attachments/...`. This needs no restart, no env change, no code change.
- The TTS scripts under `/home/claude/jarvis-tts` still write their working output there; copy the resulting `.wav` into `data/attachments` before attaching (jarvis-tts itself is NOT allow-listed).
- System defaults still allowed for internal use: the codex `generated_images` dir and the system temp dir (used for auto-generated screenshots). Only `data/attachments` is the place WE deliberately stage files to send.
- Safety net: if a file is ever referenced from outside an allowed folder, delivery now appends a visible "전송 실패 (사유)" notice instead of silently dropping it.
- **Auto-cleanup (2026-06-12 user decision):** prefer `/home/claude/EJClaw/data/attachments/generated/` for files we send. A user crontab entry sweeps that subfolder every 15 min and deletes files older than 60 min (`find .../data/attachments/generated -type f -mmin +60 -delete`). The sweep is scoped to `generated/` only, so system-managed attachments under `data/attachments` (e.g. `paired-turn-outputs/`, dashboard copies) are never touched. If a sent file must persist longer than ~1h, put it directly under `data/attachments/` (not `generated/`).
## Stored credentials
Shared credentials live at `/home/claude/.config/ejclaw/secrets.json` (chmod 600, owner-only). Read with the Read tool when a session in any channel asks to "저장해둔 계정토큰으로 로그인" or otherwise needs a registered token.
Schema: `credentials.<host>.{type, host, token, note, added_at}`.
Currently stored:
- `git.tkrmagid.kr` — Gitea personal access token. Use via `Authorization: token <value>` header, or embed in HTTPS URL as `https://<user>:<token>@git.tkrmagid.kr/...`. For `git clone/push`, prefer the URL form or `git -c http.extraHeader="Authorization: token <value>" clone ...`. Do not paste the raw token into chat replies.
- `sudo` — local sudo password for the `claude` user on this host. Use via `echo "$PW" | sudo -S <cmd>` (read the password from `credentials.sudo.password` with the Read tool, then pipe). Do not paste the raw password into chat replies.
- `discord.com/bot/testbot` — Discord test bot (테스트봇, client/app ID `1538122882528321536`), stored 2026-08-15 for use across all chats. Read the token from `credentials["discord.com/bot/testbot"].token` in secrets.json with the Read tool (token is NOT written here). Use as `Authorization: Bot <token>` against `https://discord.com/api/v10`, or as the bot token for a discord.js/gateway login. Do not paste the raw token into chat replies. Token was chat-exposed on save; if used for anything sensitive/long-term, rotate it in the Discord Developer Portal first. Re-ask the user if Discord returns 401.
- Other entries (each has its own `note` with usage rules — read it before use): `docker5.ssh` (root password fallback for .5; prefer key auth), `kis.koreainvestment` + `dart.fss.or.kr` (stock-adviser APIs), `huggingface.co` (read token), `anthropic.api` (stock-adviser API key with a monthly budget cap; also usable to list models via `GET /v1/models`), `tossinvest.com` (IP-allowlisted), `washing_machine_app.keystore`, `api.smartthings.com`, `discord.com/oauth/virtual-stock-site-bot`, `youtube.oauth/music_bot_v2`, and per-repo GitHub PATs `github.com/Kacarong/{style-ai-site,flight-tracker,stock-adviser,virtual-stock-site-bot,cgv-macro}` (use only via ephemeral push URLs, never persist in `.git/config`).
To add a credential: edit `secrets.json`, append a new entry under `credentials`, and add the key + intended use to this list. Never paste raw secret values into chat or into this file.
Git remotes on git.tkrmagid.kr no longer embed the token in the URL (cleaned 2026-10-01 in EJClaw, music-bot-test, tts_site, voice_copy_work, vp_build, groups/mc_chat_answer_mod, groups/mc_domain_proxy, groups/tts_bot). Each repo has `credential.https://git.tkrmagid.kr.helper=/home/claude/.config/ejclaw/git-credential-tkrmagid`, which serves the `git.tkrmagid.kr` token from secrets.json (keeps the URL's username, default `claude-bot`). For a new clone: use a token-free URL and run `git config --local credential.https://git.tkrmagid.kr.helper /home/claude/.config/ejclaw/git-credential-tkrmagid`. Rotating the token now only needs a secrets.json edit.
## User communication preferences
- 2026-05-27: 모든 채팅에서 사용자에게 답변할 때 기본적으로 존댓말을 사용한다. 특별히 반말을 요청받지 않는 한 반말로 답변하지 않는다.
## Room mode policy
All rooms default to `tribunal` (paired) mode. Owner (claude-code) runs the work, the reviewer (codex, `REVIEWER_AGENT_TYPE=codex`) verifies, and the arbiter (claude-code) breaks deadlocks. New rooms registered via `bun setup/index.ts --step register` are `tribunal` by default and get their paired workspace (`work_dir`) provisioned at registration. A room that ends up tribunal with `work_dir = NULL` (e.g. created through another path) is now self-healed on its first turn (fix c7a8a43, 2026-09-19); before that fix such rooms silently dropped follow-up messages.
If the user in a channel says any of these — "클로드 사용하지 말자", "paired 모드 끄자", "리뷰어 끄자", "이 방은 single 로 바꿔줘" — switch that channel back to `single` mode. Use:
```bash
bun -e "import { initDatabase, setExplicitRoomMode } from './src/db.js'; initDatabase(); setExplicitRoomMode('<chatJid>', 'single');"
```
The reverse phrase ("paired 켜자", "리뷰어 다시 켜자") flips it back to `'tribunal'`. Acknowledge the change in chat and confirm the new mode.
## Room deregistration (채팅 등록 해제 + 데이터 정리)
When the user says any of — "채팅 아이디 등록해제 해줘 <id>", "이 채널 등록 해제해줘", "채널 등록 해제" (with a channel id / JID) — the standing behavior (user decision 2026-08-18) is: fully unregister the room AND purge all managed data, keeping ONLY the chat channel record and its message history, then hot-reload (SIGHUP, NOT a full restart — see below) so the channel goes unregistered in the live process.
Use the canonical script (keeps `chats` + `messages`, removes everything else — room_settings/role/skill overrides, paired tasks/turns/attempts/outputs/reservations/leases/projects/handoffs, work_items, scheduled_tasks, task_run_logs, sessions, router cursor, and the on-disk `groups/<folder>`, `data/workspaces/<folder>`, `data/sessions/<folder>`, `data/ipc/<folder>` including any git worktree):
```bash
cd /home/claude/EJClaw
bun scripts/deregister-room.ts --dry-run <channelId...> # preview what will be removed
bun scripts/deregister-room.ts <channelId...> # execute the purge
```
Accepts bare Discord channel IDs (auto-prefixed to `dc:`) or full JIDs; multiple ids allowed. Safety built in: a group folder still used by another room is NOT deleted on disk (only that chat's DB rows); `--dry-run` writes nothing.
Applying to the live process is now automatic: after the purge the script calls `signalEjclawReload()`, which sends `SIGHUP` to the running service's main PID → `runtimeState.reloadRoomBindings()`, so the room drops from live bindings with NO full restart. Do NOT run a plain `systemctl restart` from inside a turn (it kills the agent mid-command and can't be confirmed). Verify by the `Room bindings reloaded` log or `getRegisteredGroup(jid)` → undefined. Always take a quick DB backup first for irreversible purges: `cp store/messages.db /home/claude/ejclaw-db-backup-$(date +%Y%m%d-%H%M%S).db`.
## Channel registration + making it live (no false restarts) — MANDATORY
The agent runs INSIDE `ejclaw.service`, so a plain `systemctl --user restart ejclaw.service` from within a turn kills the agent mid-command: the restart is unreliable and you CANNOT confirm it in the same turn. Never claim a restart/reload happened without evidence — the repeated "재시작 됐다고 했는데 안 됨" complaints came from claiming success without verifying.
When you register (or deregister) a channel via a DB write (bun script / `setup --step register`) from a room, the running process does NOT see it until its in-memory room bindings are reloaded. Preferred way — hot-reload, NO full restart, agent survives so you can verify in the same turn:
```bash
kill -HUP "$(systemctl --user show -p MainPID --value ejclaw.service)"
```
`src/index.ts` handles `SIGHUP` → `runtimeState.reloadRoomBindings()` (re-reads ONLY room bindings from the DB; message cursors/sessions untouched, so no re-processing). Sending HUP to the MainPID only (not the whole cgroup) leaves agent subprocesses alive. After it, VERIFY before reporting: look for the `Room bindings reloaded` log with the new `groupCount`, or check `getRegisteredGroup(jid)` resolves. Standing user rule: registering a channel must be followed immediately by this live-reload (this is the "등록하면 자동 재시작/반영" the user asked for) — do it once, verify, then report.
Only when a change truly needs a full process restart (e.g. code/env changes), use the detached form so it survives the agent's death, and verify in a SEPARATE scheduled task (new PID / start time) — do it exactly once:
```bash
systemd-run --user --collect --unit="ejclaw-restart-$$" systemctl --user restart ejclaw.service
```
## washing_machine_app (EasyAppliance Android) deployment
- 2026-08-06 user standing rule: for the washing_machine_app project, ALWAYS deploy after a change and deliver via in-app update — do not just commit. Every meaningful change ends with a published Gitea release so the in-app updater offers it.
- Build env: default JDK on this host is Java 25, which the Gradle Kotlin compiler can't parse ("IllegalArgumentException: 25.0.3"). Build with `export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64`. Android SDK at `/home/claude/android-sdk` (build-tools 35.0.0 has aapt/apksigner).
- Release process: bump `versionCode`/`versionName` defaults in `app/build.gradle.kts` (also passable via `-PverCode -PverName`), `./gradlew assembleRelease` (auto-signed via `keystore.properties` → keystore at `/home/claude/.config/ejclaw/washing_machine_app.keystore`, release cert SHA-256 `56246bd2…`; every release MUST use this same key or in-place update breaks). Then create a Gitea release + upload the APK asset named `EasyAppliance-v<ver>.apk`.
- Release host: Gitea `git.tkrmagid.kr`, repo `tkrmagid/washing_machine_app`. Use the stored `git.tkrmagid.kr` token (secrets.json). API: `POST /api/v1/repos/tkrmagid/washing_machine_app/releases` then `POST /releases/<id>/assets?name=...` with the APK. The app's `UpdateRepository` reads releases via the public API and offers any release whose tag (minus `v`) is newer than installed. Latest published: v0.3.9 (verCode 12) as of 2026-08-06. Token now stored encrypted (AndroidKeyStore AES/GCM via TokenCrypto); allowBackup=false, usesCleartextTraffic=false.
- No git remote is configured in the owner worktree; the Gitea release (APK) is the delivery mechanism, separate from any source push.
- Real-device tuning: the user's SmartThings Personal Access Token is stored in secrets.json under `api.smartthings.com` (Bearer, `https://api.smartthings.com/v1`). Use it to read actual device capabilities/supported values so preset/status mappings stay accurate. Their real devices: 세탁기 `b9b0b550-…` (water temp enum none/cold/20/30/40/60/90; spin `custom.washerSpinLevel`; rinse `custom.washerRinseCycles`; NO manual water level — auto), 벽걸이/스탠드 에어컨 (mode cool/dry/wind/aIComfort; fan auto/1/2/3/4/max; setpoint 16–30). SmartThings commands are type-strict: appliance enum args (washer temp "40", fan "1", rinse "3") are STRINGS; only setpoint-style commands are NUMBERS — `SmartThingsDeviceRepository.encodeArg` handles this via a NUMERIC_COMMANDS allowlist.
## Git backups
- 2026-05-27 10:53 KST: EJClaw 정상 동작 상태를 git commit `1509108` (`backup current stable ejclaw state`)로 백업했다. "이번 요청만 리뷰어 사용" 기능 작업은 이 백업 커밋 이후에 시작한 변경이다.
## Per-service Claude credential isolation (user changes, verified 2026-06-23)
The user reworked credential handling so each service reads its OWN Claude credentials file instead of all sharing `~/.claude/.credentials.json`. Goal: run each service on a different Claude account.
- Resolver: `src/claude-credentials-path.ts` — `getClaudeCredentialsPath(accountIndex, {allowHomeFallback})` and `hasExplicitClaudeCredentialsPath()`. Order: account 0 → `CLAUDE_CREDENTIALS_PATH`; account 1+ → `CLAUDE_ACCOUNTS_DIR/<n>/.credentials.json`; else home fallback (`~/.claude` or `~/.claude-accounts/<n>`). The same module exists in both EJClaw and KAclaw.
- Wired into: `setup/login.ts` (writes to `CLAUDE_CREDENTIALS_PATH`; PKCE state file is now per-path-hashed `ejclaw-claude-login-<hash>.json` so different-account logins don't collide), `src/token-refresh.ts`, `src/claude-usage.ts`, `src/agent-runner-environment.ts` (pre-syncs creds into each session `.claude/.credentials.json`), `src/runtime-inventory.ts`.
- Credential paths in use:
- claude CLI: `~/.claude/.credentials.json`
- EJClaw: `/home/claude/EJClaw/data/claude/.credentials.json` (set via `CLAUDE_CREDENTIALS_PATH` in `EJClaw/.env`)
- KAclaw: `/home/claude/KAclaw/data/claude/.credentials.json` (set via `CLAUDE_CREDENTIALS_PATH` in `KAclaw/.env`)
- Relevant env flags (EJClaw and KAclaw both): `CLAUDE_USAGE_USE_HOME_CREDENTIALS=false`, `CLAUDE_TOKEN_REFRESH_USE_CREDENTIALS=true`, `CLAUDE_TOKEN_REFRESH_USE_HOME_CREDENTIALS=false`. The token-refresh loop only runs when `CLAUDE_TOKEN_REFRESH_USE_CREDENTIALS=true` OR `*_USE_HOME_CREDENTIALS=true`.
- To re-auth ONE service: run its login flow with that service's `CLAUDE_CREDENTIALS_PATH` exported, then send the new `code#state`. Do NOT copy one service's `.credentials.json` to another path — that collapses them onto one token + one shared refresh token, and when both refresh independently the refresh token rotates and the other side breaks with `invalid_grant`.
- Account-identity reality (verified via `https://api.anthropic.com/api/oauth/profile`, 2026-06-23 ~19:35 KST): despite the intent, all three paths still resolved to the SAME account `tkrmagid@gmail.com` (Max). CLI and KAclaw even held byte-identical access+refresh tokens (from an earlier `~/.claude`→KAclaw copy); EJClaw held a different token of the same account. Before assuming the services are on distinct accounts, verify the actual account by calling the profile API for each path's accessToken (compare `account.email`), not just by checking that the file paths differ.
- KAclaw is NOT a git repo (no `.git`), so its source changes can't be diffed/version-controlled. 2026-06-23 dashboard usage-row fix (utilization of exactly 1 rendering as 100%, `> 1` → `>= 1` in `dashboard-usage-rows.ts`) lives only in the KAclaw working tree. EJClaw's copy of that file may carry the same `> 1` pattern — worth checking if the EJClaw dashboard ever shows a window at exactly 1%.
- UPDATE 2026-06-23 ~21:26 KST: KAclaw was re-logged onto a DIFFERENT account at the user's request. Now EJClaw = `tkrmagid@gmail.com` (Max) and KAclaw = `hjj100411@gmail.com` (NOT Max — has_claude_max=false, so Pro/Free, lower limits). Done via KAclaw's own `setup/index.ts --step login` with `CLAUDE_CREDENTIALS_PATH=/home/claude/KAclaw/data/claude/.credentials.json` exported, then updated KAclaw `.env` CLAUDE_CODE_OAUTH_TOKEN(S) to the new access token and restarted `kaclaw.service`. claude CLI (`~/.claude`) is still `tkrmagid@gmail.com`. So EJClaw and KAclaw are now genuinely different accounts; verify with the profile-API email check if in doubt.
## Codex (reviewer) credential isolation (2026-07-22)
EJClaw's paired-room **reviewer** runs as Codex (gpt-5.5) with ChatGPT-OAuth auth. It shared `~/.codex/auth.json` with other consumers (manual `codex` CLI, KAclaw same-home), so refreshes rotated the token and the stale copy died with "access token could not be refreshed because your refresh token was already used." Every reviewer turn then failed → task ended `reviewer_codex_unavailable` and escalated straight to the arbiter, so chats showed owner + arbiter but the reviewer never posted. Diagnose via `paired_turn_attempts.last_error` in `store/messages.db`.
Fix (mirrors the Claude per-service isolation above): EJClaw now uses a dedicated Codex account dir `~/.codex-accounts/0/` instead of shared `~/.codex`. `src/codex-token-rotation.ts` `initCodexTokenRotation()` loads numbered dirs under `~/.codex-accounts/` as the rotation pool and ignores the `~/.codex` fallback when the dir has entries; EJClaw leases + writes-back refreshes to `~/.codex-accounts/0/auth.json` (`syncCodexSessionAuthBack`), isolated from the CLI and KAclaw.
- **Activation needs an ejclaw restart** — `initCodexTokenRotation()` runs once at process start (`src/index.ts`), so a running ejclaw keeps the old cached account until `systemctl --user restart ejclaw`. Do NOT restart from inside a live agent turn (the agent is a descendant).
- **Re-auth runbook (the procedure CHANGED):** refresh the reviewer's Codex login into the isolated dir, NOT default `~/.codex`:
`CODEX_HOME=/home/claude/.codex-accounts/0 /home/claude/EJClaw/node_modules/.bin/codex login --device-auth`
then open the printed `https://auth.openai.com/codex/device` URL and enter the one-time code. A plain `codex login` writes to `~/.codex` and will NOT update the account EJClaw uses. Account: ChatGPT `account_id 7e40ecd8-461b-49bd-bbe4-cdc6c7dc68cf`.
- Verify: `cd /home/claude/EJClaw && bun -e 'import {initCodexTokenRotation,getActiveCodexAuthPath} from "./src/codex-token-rotation.ts"; initCodexTokenRotation(); console.log(getActiveCodexAuthPath())'` → should print `/home/claude/.codex-accounts/0/auth.json`.
## EJClaw / KAclaw model + runtime versions (updated 2026-10-01)
Current state:
- EJClaw: `CLAUDE_MODEL=claude-opus-5-5` (Claude Opus 5.5), `CLAUDE_EFFORT=high`, `CLAUDE_THINKING=adaptive`. Runner SDK `@anthropic-ai/claude-agent-sdk` 0.3.286 → bundled Claude Code CLI 2.1.286 (pinned in `runners/agent-runner/package.json`). Reviewer: Codex `gpt-5.5` via `@openai/codex` 0.159.3 (pinned in `runners/codex-runner/package.json`). `.env` backup before the 5.5 switch: `/home/claude/ejclaw-env-backup-20261001-140746`.
- KAclaw: `CLAUDE_MODEL=claude-opus-5-5` since 2026-10-01 (account hjj100411, Pro). Its runner is a standalone package (no workspaces): SDK bumped 0.2.114 → 0.3.286 in `runners/agent-runner` (CLI 2.1.114 rejected Opus 5.5). KAclaw is not a git repo; pre-change backup (runner package.json, bun.lock, dist, node_modules, .env, expired credentials) at `/home/claude/kaclaw-backup-20261001-144805/`.
- KAclaw OAuth: its refresh token expired 2026-09-30 22:43 (`invalid_grant: Refresh token expired`; refresh tokens have an absolute lifetime), the access token died 23:38 and KAclaw silently could not answer until re-login. Re-login = `cd /home/claude/KAclaw && CLAUDE_CREDENTIALS_PATH=/home/claude/KAclaw/data/claude/.credentials.json bun setup/index.ts --step login` (prints URL) → account owner logs in → re-run with `--code <code#state>` → update `CLAUDE_CODE_OAUTH_TOKEN(S)` in KAclaw `.env` → `systemctl --user restart kaclaw` (safe from an EJClaw turn).
- Testing a model "with a service's account": the Claude Code CLI ignores `CLAUDE_CREDENTIALS_PATH`; without `CLAUDE_CODE_OAUTH_TOKEN` it silently uses `~/.claude` (tkrmagid Max). Pass `CLAUDE_CODE_OAUTH_TOKEN=<that service's access token>` explicitly, or the test proves nothing about that account.
Model upgrade checklist (both incidents below were caused by skipping it):
1. Get the exact model ID from the models API, never by guessing from the marketing name: `curl -s https://api.anthropic.com/v1/models -H "x-api-key: <anthropic.api key>" -H "anthropic-version: 2023-06-01"`. Valid 5.x IDs as of 2026-10-01: `claude-opus-5-5`, `claude-sonnet-5-5`, `claude-opus-5`, `claude-sonnet-5`, `claude-fable-5-1`, `claude-fable-5`. Writing `5.0` as `claude-opus-5-0` is invalid — that typo broke every KAclaw channel 2026-09-17 → 09-21.
2. Before editing `.env`, run a real query through the service's own SDK + bundled binary with that service's access token in `CLAUDE_CODE_OAUTH_TOKEN` (`query({model, pathToClaudeCodeExecutable: <runner>/node_modules/@anthropic-ai/claude-agent-sdk-linux-x64/claude})`; `CLAUDE_CREDENTIALS_PATH` alone is ignored by the CLI and silently falls back to `~/.claude`). Newer models can require a newer Claude Code CLI: Opus 5.5 is rejected by CLI 2.1.153 with `400 ... version 2.1.280 or newer is required`, so the SDK had to be bumped first.
3. Opus/Sonnet 5.x support only `adaptive` thinking — keep `CLAUDE_THINKING=adaptive` (`enabled` + budget fails on 5.x).
4. Apply with the detached `systemd-run … systemctl --user restart` form and verify the new PID + zero `does not support this model` / `issue with the selected model` log lines.
Gotchas found 2026-10-01:
- **Stale nested `node_modules` can shadow the lockfile version.** `runners/codex-runner/node_modules/@openai/codex` held an untracked 0.124.0 copy from 2026-04-26, so the reviewer kept running 0.124 no matter what the pin said (that old CLI failed every run with `failed to decode models response: unknown variant max` and dumped the whole models payload into the journal). It was removed (backup `/home/claude/.tmp/stale-codex-backup/`). After any runner dependency bump, verify what the runner really loads: `cd runners/codex-runner/dist && bun -e "const p=Bun.resolveSync('@openai/codex/package.json', process.cwd()+'/index.js'); console.log(p, require(p).version)"` — it must be the root `node_modules` path.
- **One Codex account = one Codex run at a time.** The per-account auth lease is exclusive (protects refresh-token rotation). Two rooms needing Codex at once used to fail the second run instantly ("Codex rotation pool unavailable") and close the task as `reviewer_codex_unavailable` — the review was silently skipped (3× on 2026-09-28). Since fix 30ba510 the second run waits up to 15 min for the lease; a dead/rate-limited pool still fails fast. Adding a second Codex account under `~/.codex-accounts/1/` would remove the waiting.
- The service runs `bun src/index.ts`, so root `dist/` staleness does not matter; runner `dist/` (spawned by path) does — rebuild with `bun run build:runners` after runner changes.
EJClaw fixes 2026-09-19 → 10-01 (details in each commit message):
- c7a8a43 — tribunal room with `work_dir = NULL` self-heals instead of freezing its task revision and dropping follow-ups for 10 min.
- 69ae334 — superseded run's 5s progress ticker is stopped when it loses the turn attempt (room no longer looks frozen).
- 8b68885 — claude-agent-sdk 0.3.153 → 0.3.286 for Opus 5.5.
- 30ba510 — wait for a busy Codex lease instead of skipping the review.
- a7f2e02 — @openai/codex 0.133 → 0.159.3 (+ stale 0.124 nested copy removed); simple reviewer turn 49s → 5s, 0 app-server errors.
## Windows CI 러너 / .9 중첩 가상화 (2026-09-23) — 모든 프로젝트 공용
`.9` 안에서 Windows VM 을 돌려 Windows 전용 경로를 자동 검증할 수 있다.
LiveSub 에서 처음 구축했지만 **어느 프로젝트든 그대로 재사용**하면 된다.
### 중첩 가상화 — 된다 (실측)
`.9` 는 Proxmox 위의 QEMU/KVM VM 이지만 중첩 가상화가 켜져 있다.
`/dev/kvm` 존재, `kvm_intel nested=Y`, `qemu -enable-kvm` → `kvm support: enabled`.
- 설치해 둔 것: `qemu-system-x86`, `ovmf`, `swtpm`, `swtpm-tools`, `xorriso`
- **`claude` 계정을 `kvm` 그룹에 넣었다.** `/dev/kvm` 이 `root:kvm 0660` 이라
없으면 `Could not access KVM kernel module: Permission denied` 로 죽는다.
usermod 는 재로그인해야 적용되므로 그 전에는 `sg kvm -c '...'` 로 감싼다.
- **중첩 GPU 패스스루는 불가능하다.** IOMMU 그룹 0개, 커널 cmdline 에 iommu
옵션 없음. GPU 검증은 `.9`(리눅스)에서 직접 해야 한다.
### 자원 한계 — CPU 는 절대 늘리지 말 것 (실측 근거)
물리 호스트는 **i7-11700 (8코어/16스레드), RAM 62.6GB** 다. Proxmox(`192.168.10.2`)
에는 SSH 키가 없어 접근 못 하지만, `.5` LXC 를 통해 호스트 스펙을 읽을 수 있다.
`/proc/meminfo` 는 lxcfs 가 컨테이너 한도로 가리므로
**`/sys/devices/system/node/node0/meminfo`** 를 읽어야 진짜 값이 나온다.
`.9` 의 vCPU 를 늘리면 `.5` 의 운영 컨테이너 19개가 느려진다. `.5` 응답 실측:
```
.9 한가할 때 616~628ms 기준
.9 가 4코어 사용 657~678ms +7% <- VM 100% 인 정상 상황
.9 가 8코어 전부 1487~1491ms +140% <- 이 상태를 만들면 안 된다
```
그래서 VM 은 **vCPU 4 / RAM 8GB / 디스크 80GB** 로 띄우고, 최악을 막으려면
`CPUQuota=400%` + `CPUWeight=50` 을 건다. CI 작업은 코어 4개에서 포화하므로
더 줘도 빨라지지 않는다 (파이프라인 2코어 13초 / 4코어 10초 / 8코어 11초).
### ⚠️ VM 은 반드시 `systemd-run --user --scope` 로 띄울 것
에이전트 턴에서 그냥 `qemu` 를 실행하면 VM 이 **봇의 cgroup 안에서** 돈다.
`ejclaw.service` 는 `MemoryHigh=16G` 이고 8GB VM 이 그 안에 들어가면 상한을
쳐서 봇이 OOM 으로 죽는다 (2026-07-26 에 실제로 6회 연속 OOM 전례 있음).
```bash
systemd-run --user --scope -p MemoryMax=10G -p CPUQuota=400% -p CPUWeight=50 \
--unit win-ci qemu-system-x86_64 -enable-kvm -m 8192 ...
```
`app.slice` 의 형제 scope 로 빠지는 것을 실측 확인했다.
### ISO 출처 (라이선스 키 불필요)
- **Windows 11 Enterprise LTSC 2024 평가판** (90일, 키 없이 설치·사용)
`https://go.microsoft.com/fwlink/p/?linkid=2289029` → 4.8GB
파일명 `26100.1742...CLIENT_LTSC_EVAL_x64FRE_en-us.iso` (빌드 26100 = 24H2)
LTSC 라 Store·소비자 앱이 없어 CI 러너로 오히려 유리하다.
90일 후에는 `slmgr /rearm` 으로 최대 3회 연장하거나 VM 을 다시 만든다.
- **virtio-win** (virtio 드라이버)
`https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso` → 0.9GB
- 다운로드 속도 약 12.5MB/s → 둘 합쳐 약 8분.
- 사용자 지시: **설치 끝나면 ISO 는 지워서 디스크를 회수할 것.**
### VM 구성 시 빠뜨리면 조용히 실패하는 것들
- `-device virtio-vga,max_outputs=2` — 화면 2개. 기본 `-vga std` 는 1개라
**모니터 선택 기능을 검증할 수 없다.** 게스트에 virtio-gpu 드라이버 필요.
- 오디오 장치를 안 붙이면 WASAPI 장치 열거가 0개로 실패한다.
- **자동 로그온 필수.** 러너를 서비스로 돌리면 데스크톱 세션이 없어 GUI
테스트와 스크린샷이 전부 실패한다. 자동 로그온 + 로그온 시 예약 작업.
- Windows 11 은 TPM 2.0 + SecureBoot 필요 → `swtpm` + `OVMF_CODE_4M.secboot.fd`.
### Gitea Actions 러너
- 라벨 형식 `<name>[:<schema>]`, schema 기본값 `host`.
- **라벨이 안 맞아도 작업이 실패하지 않고 기본 ubuntu 컨테이너에서 돈다.**
Windows 검증이 조용히 리눅스에서 도는 착각이 생기므로 워크플로 첫 단계에서
`RUNNER_OS` 를 확인해 막을 것.
- `shell: bash` 를 쓰지 말 것. Git for Windows 를 깔아도 `bash.exe` 는 기본
PATH 에 없다 (`Git\cmd` 만 등록). `pwsh`(PowerShell 7) 를 쓰고 winget 으로
`Microsoft.PowerShell` 을 설치해야 한다.
- GPU 러너를 여러 프로젝트가 공유하면 **`capacity: 1`** 을 반드시 확인한다.
워크플로 `concurrency` 는 같은 저장소 안에서만 막아준다.
### .9 재부팅 시 확인 (2026-09-23 기준)
`linger=yes` 이고 아래 유저 서비스가 전부 `enabled` 라 재부팅 후 자동 복구된다.
ejclaw, kaclaw, geoglobe-web/tunnel, unifi-site/tunnel, exvideo-tunnel,
wsai-voice, xmas-counter, vncserver, filter-chain. 시스템 쪽은 docker,
containerd, ollama 가 enabled. 재부팅 전에 이 목록을 다시 확인할 것.
## .5 Docker host + remote docker context (2026-07-21)
A second LAN machine runs Docker workloads; any Claude channel on .9 can drive it.
- Identity: `192.168.10.5` — a Proxmox **LXC container** hostnamed `docker`, Debian 13, Docker 29.3.0, root-only shell, **no GPU**. This host (.9, `192.168.10.9`, "claude") is a QEMU/KVM VM with the RTX 5050 passed through — so .9 has the GPU, .5 does not.
- SSH: **passwordless key auth is installed** (`claude@.9` pubkey in `root@.5:~/.ssh/authorized_keys`) → `ssh root@192.168.10.5` needs no password. Fallback root password in `secrets.json` under `docker5.ssh`.
- **Docker context** `docker5` (`ssh://root@192.168.10.5`) exists on .9 and is the **active default context**, so a plain `docker ...`/`docker compose ...` on .9 runs on .5. Use `docker --context default ...` (or `docker context use default`) for .9's own local docker (where the GPU Ollama service for B would run); switch back with `docker context use docker5`.
- .5 runs ~19 production containers — do NOT disturb: projects `bot` (mamil, memil, music1/2/3, random), `site` (make_video_site, minecraft_launcher_site, signature), `virtual-stock-site-bot` (api/web/worker), `mc_domain_proxy` (mc-filter api/frontend/nginx/proxy), `act_runner`, `portainer`, `spotify-tokener`. Compose files under `root@.5:/root/other/*/docker-compose.yml`.
- GPU-from-.5: since .5 has no GPU device, GPU work for .5 containers must go over the network to .9 (e.g. Ollama on .9 `--gpus all` bound to `0.0.0.0:11434`, .5 calls `http://192.168.10.9:11434`). In-container CUDA (e.g. faster-whisper `device=cuda`) will NOT work on .5.
---
- All rooms are `tribunal` (owner + reviewer + arbiter) unless the user asks for single mode in that room ("리뷰어 끄자", "single로 바꿔줘"). Commands: `ops/ejclaw-ops.md`.
- "채널 등록 해제 <id>" means a full purge with `bun scripts/deregister-room.ts` (dry-run first, DB backup first). Details: `ops/ejclaw-ops.md`.
- A short stop-only message ("그만해", "멈춰", "중단", "stop") stops the room's running work; this is built in.
# CLAUDE.md