diff --git a/.claude/skills/add-voice-transcription/SKILL.md b/.claude/skills/add-voice-transcription/SKILL.md index c3c0043..431e3cd 100644 --- a/.claude/skills/add-voice-transcription/SKILL.md +++ b/.claude/skills/add-voice-transcription/SKILL.md @@ -1,144 +1,90 @@ --- name: add-voice-transcription -description: Add voice message transcription to NanoClaw using OpenAI's Whisper API. Automatically transcribes WhatsApp voice notes so the agent can read and respond to them. +description: Add voice message transcription to NanoClaw using Groq Whisper (fast, free) with OpenAI fallback. Works on Discord and WhatsApp. --- # Add Voice Transcription -This skill adds automatic voice message transcription to NanoClaw's WhatsApp channel using OpenAI's Whisper API. When a voice note arrives, it is downloaded, transcribed, and delivered to the agent as `[Voice: ]`. +This skill adds automatic voice message transcription to NanoClaw. When a voice note arrives, it is downloaded, transcribed, and delivered to the agent as `[Voice message transcription]: `. + +**Provider priority:** Groq Whisper (fast, free) > OpenAI Whisper (fallback). ## Phase 1: Pre-flight -### Check if already applied +### Discord + +Voice transcription is built into `src/channels/discord.ts`. No code changes needed — just configure API keys (Phase 3). + +### WhatsApp Check if `src/transcription.ts` exists. If it does, skip to Phase 3 (Configure). The code changes are already in place. -### Ask the user - -Use `AskUserQuestion` to collect information: - -AskUserQuestion: Do you have an OpenAI API key for Whisper transcription? - -If yes, collect it now. If no, direct them to create one at https://platform.openai.com/api-keys. - -## Phase 2: Apply Code Changes - -**Prerequisite:** WhatsApp must be installed first (`skill/whatsapp` merged). This skill modifies WhatsApp channel files. - -### Ensure WhatsApp fork remote - -```bash -git remote -v -``` - -If `whatsapp` is missing, add it: - -```bash -git remote add whatsapp https://github.com/qwibitai/nanoclaw-whatsapp.git -``` - -### Merge the skill branch +If not, merge the skill branch: ```bash +git remote add whatsapp https://github.com/qwibitai/nanoclaw-whatsapp.git 2>/dev/null git fetch whatsapp skill/voice-transcription git merge whatsapp/skill/voice-transcription -``` - -This merges in: -- `src/transcription.ts` (voice transcription module using OpenAI Whisper) -- Voice handling in `src/channels/whatsapp.ts` (isVoiceMessage check, transcribeAudioMessage call) -- Transcription tests in `src/channels/whatsapp.test.ts` -- `openai` npm dependency in `package.json` -- `OPENAI_API_KEY` in `.env.example` - -If the merge reports conflicts, resolve them by reading the conflicted files and understanding the intent of both sides. - -### Validate code changes - -```bash npm install --legacy-peer-deps npm run build -npx vitest run src/channels/whatsapp.test.ts ``` -All tests must pass and build must be clean before proceeding. +## Phase 2: Configure -## Phase 3: Configure +### Get API key -### Get OpenAI API key (if needed) +**Groq (recommended — fast + free):** -If the user doesn't have an API key: - -> I need you to create an OpenAI API key: +> 1. Go to https://console.groq.com +> 2. Sign up (no credit card needed) +> 3. Create an API key (starts with `gsk_`) > +> Free tier: 2,000 requests/day, 8 hours of audio/day. Uses `whisper-large-v3-turbo` at ~200x real-time speed. + +**OpenAI (fallback):** + > 1. Go to https://platform.openai.com/api-keys -> 2. Click "Create new secret key" -> 3. Give it a name (e.g., "NanoClaw Transcription") -> 4. Copy the key (starts with `sk-`) +> 2. Create a key (starts with `sk-`) > -> Cost: ~$0.006 per minute of audio (~$0.003 per typical 30-second voice note) - -Wait for the user to provide the key. +> Cost: ~$0.006 per minute of audio. Requires funded account. ### Add to environment Add to `.env`: ```bash -OPENAI_API_KEY= +GROQ_API_KEY=gsk_... # Primary (fast, free) +OPENAI_API_KEY=sk-... # Fallback (optional if Groq is set) ``` -Sync to container environment: - -```bash -mkdir -p data/env && cp .env data/env/env -``` - -The container reads environment from `data/env/env`, not `.env` directly. - ### Build and restart ```bash npm run build -launchctl kickstart -k gui/$(id -u)/com.nanoclaw # macOS -# Linux: systemctl --user restart nanoclaw +systemctl --user restart nanoclaw nanoclaw-codex # Linux +# macOS: launchctl kickstart -k gui/$(id -u)/com.nanoclaw ``` -## Phase 4: Verify +## Phase 3: Verify -### Test with a voice note +Send a voice note in any registered chat. The agent should receive it as `[Voice message transcription]: `. -Tell the user: - -> Send a voice note in any registered WhatsApp chat. The agent should receive it as `[Voice: ]` and respond to its content. - -### Check logs if needed +### Check logs ```bash -tail -f logs/nanoclaw.log | grep -i voice +tail -f logs/nanoclaw.log | grep -iE "transcri|audio" ``` Look for: -- `Transcribed voice message` — successful transcription with character count -- `OPENAI_API_KEY not set` — key missing from `.env` -- `OpenAI transcription failed` — API error (check key validity, billing) -- `Failed to download audio message` — media download issue +- `Audio transcribed + cached` with `provider: "groq"` and `elapsed` — success +- `Transcription cache hit` — second service read from cache (no duplicate API call) +- `no transcription API key` — neither `GROQ_API_KEY` nor `OPENAI_API_KEY` set +- `groq Whisper 4xx` — check key validity ## Troubleshooting -### Voice notes show "[Voice Message - transcription unavailable]" +**No transcription:** Check `GROQ_API_KEY` (or `OPENAI_API_KEY`) is set in `.env`. Restart service after changes. -1. Check `OPENAI_API_KEY` is set in `.env` AND synced to `data/env/env` -2. Verify key works: `curl -s https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY" | head -c 200` -3. Check OpenAI billing — Whisper requires a funded account +**Slow transcription:** Verify `provider: "groq"` in logs. If it shows `openai`, the Groq key may be missing or invalid. -### Voice notes show "[Voice Message - transcription failed]" - -Check logs for the specific error. Common causes: -- Network timeout — transient, will work on next message -- Invalid API key — regenerate at https://platform.openai.com/api-keys -- Rate limiting — wait and retry - -### Agent doesn't respond to voice notes - -Verify the chat is registered and the agent is running. Voice transcription only runs for registered groups. +**Agent doesn't respond to voice notes:** Verify the chat is registered and the agent is running. Voice transcription only runs for registered groups. diff --git a/.claude/skills/setup/SKILL.md b/.claude/skills/setup/SKILL.md index 703fc1d..02e9a87 100644 --- a/.claude/skills/setup/SKILL.md +++ b/.claude/skills/setup/SKILL.md @@ -99,6 +99,12 @@ AskUserQuestion: Do you also want to use Codex (OpenAI) agents? If yes: Tell user to add `OPENAI_API_KEY=` (or `CODEX_OPENAI_API_KEY=`) to `.env`. The Codex runner will use this key. +### Voice Transcription (Optional) + +AskUserQuestion: Do you want voice message transcription in Discord/WhatsApp? + +If yes: Tell user to get a free API key at https://console.groq.com (no credit card needed) and add `GROQ_API_KEY=` to `.env`. Uses Groq Whisper (whisper-large-v3-turbo, ~200x faster than OpenAI). Falls back to OpenAI Whisper (`OPENAI_API_KEY`) if Groq key is not set. + ## 5. Install Skills Marketplace Register and install the NanoClaw skills marketplace plugin so all feature skills (channel integrations, add-ons) are available: @@ -188,6 +194,8 @@ Tell user to test: send a message in their registered chat. Show: `tail -f logs/ **Codex agent not responding:** Ensure `OPENAI_API_KEY` is set in `.env` and `codex` CLI is installed globally (`npm install -g @openai/codex`). The runner needs `codex` in PATH. +**Voice transcription not working:** Check `GROQ_API_KEY` (primary) or `OPENAI_API_KEY` (fallback) is set in `.env`. Test: send a voice message in Discord and check `logs/nanoclaw.log` for `Audio transcribed` entries with `provider` and `elapsed` fields. + **No response to messages:** Check trigger pattern. Main channel doesn't need prefix. Check DB: `npx tsx setup/index.ts --step verify`. Check `logs/nanoclaw.log`. **Channel not connecting:** Verify the channel's credentials are set in `.env`. Channels auto-enable when their credentials are present. For WhatsApp: check `store/auth/creds.json` exists. For token-based channels: check token values in `.env`. Restart the service after any `.env` change. diff --git a/CLAUDE.md b/CLAUDE.md index 3769c42..24daf11 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,7 +13,7 @@ Two systemd services (`nanoclaw`, `nanoclaw-codex`) share the same codebase but | `src/index.ts` | Orchestrator: state, message loop, agent invocation | | `src/agent-runner.ts` | Spawns agent processes, manages env/sessions/skills | | `src/token-refresh.ts` | OAuth auto-refresh + session directory sync | -| `src/channels/discord.ts` | Discord channel (8s typing refresh) | +| `src/channels/discord.ts` | Discord channel (8s typing refresh, Groq/OpenAI Whisper transcription) | | `src/ipc.ts` | IPC watcher and task processing | | `src/router.ts` | Message formatting and outbound routing | | `src/config.ts` | Trigger pattern, paths, intervals | @@ -68,3 +68,11 @@ Codex runner uses `codex app-server` JSON-RPC (not `codex exec`): - `turn/steer` for mid-execution message injection (IPC polling during turn) - `approvalPolicy: "never"` + `sandbox: "danger-full-access"` for bypass - Per-group: model (`CODEX_MODEL`), effort (`CODEX_EFFORT`), MCP servers via `config.toml` + +## Voice Transcription + +Audio attachments in Discord are transcribed via Groq Whisper (primary) or OpenAI Whisper (fallback): +- `GROQ_API_KEY` — Groq `whisper-large-v3-turbo`, ~200x real-time, free tier (console.groq.com) +- `OPENAI_API_KEY` — OpenAI `whisper-1`, fallback if Groq key not set +- Shared file cache (`cache/transcriptions/`) deduplicates across both services +- `.pending` file coordination prevents duplicate API calls diff --git a/README.md b/README.md index 032cec5..27aaae1 100644 --- a/README.md +++ b/README.md @@ -149,6 +149,9 @@ claude login # Codex — set API key export OPENAI_API_KEY=sk-... + +# Groq — for fast voice transcription (free at console.groq.com) +export GROQ_API_KEY=gsk_... ``` ### 3. Environment Variables @@ -161,6 +164,7 @@ DISCORD_BOT_TOKEN= # Claude Code Discord bot token ASSISTANT_NAME=claude # Bot trigger name (@claude) ANTHROPIC_API_KEY= # Or use OAuth (claude login) OPENAI_API_KEY= # For Codex +GROQ_API_KEY= # For voice transcription (Groq Whisper, fast + free) ``` For dual-service setup, create `.env.codex` for Codex-specific overrides: