Compare commits

..

54 Commits

Author SHA1 Message Date
javis-bot
2a95f58a0e chore(app): version 0.1.31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:15:23 +09:00
javis-bot
81d1e26f61 fix(voice): keep the first packets of a speaker until Discord names them
Discord names a speaker's SSRC in the Speaking op only the first time
they speak, and the op arrives after their first UDP packets. songbird
0.6.1 dropped the DAVE packets it could not decrypt yet, so the first
~100 ms of the first thing each person said after a join were lost
(live 2026-11-18: "자비스 서울 날씨 어때" arrived as "아비 서울날씨
어때", and the first call after the 01:20 start as "아비스").

songbird is vendored in app/vendor/songbird with one patch: packets of
an unnamed SSRC are held (1 s, 50 packets each, 32 SSRCs) and processed
in arrival order once it is named, and the Speaking event is sent before
the SSRC is named so handlers know the speaker first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:15:07 +09:00
javis-bot
88ee5ec2cb feat(weather): the app's log names the service that gave the weather
The source line (KMA, open-meteo or met.no for now; KMA and open-meteo
day counts for a forecast) only went to debug_log, which is off by
default and muted while the worker's brain answers, so a live answer
such as "흐림" could not be traced to its service. It now goes to the
llm log tab without the place or the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:10:06 +09:00
javis-bot
2e90c47a95 chore(app): version 0.1.30
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:21:46 +09:00
javis-bot
c7655c58f4 feat(weather): detail on request, 소나기 in 단기예보, weekday counts in the schema
- detail (only when the user asks for more detail) adds the humidity, the
  wind in m/s and the rain amount: now the last hour's rain (KMA RN1,
  open-meteo precipitation), a 단기예보 day its mean humidity and strongest
  wind, an open-meteo day its strongest wind and rain total.
- 단기예보's PTY 4 (소나기) was missing, so a showery afternoon under a
  clear sky came out as 맑음.
- days_ahead's description lists each coming weekday's count from today:
  with the new argument qwen3:8b counted a Wednesday's weekend as 2.

Evals: evals/test_weather_detail.py, test_weather_forecast_days.py,
test_weather_district_place.py (qwen3:8b, 27/27 over three runs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:21:46 +09:00
javis-bot
b4f8f71dd2 chore(app): version 0.1.29
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:08:15 +09:00
javis-bot
9ba26c5bac feat(weather): districts such as 부산 연제구 found on OpenStreetMap
open-meteo's geocoder knows cities only, in Latin spellings, and matches a
non-Latin name loosely: "부산" gave a village called Pusan 140 km from
Busan. A non-Latin name now goes to OpenStreetMap's Nominatim first (any
language, down to districts), and a Latin name open-meteo does not know
goes there before the LLM romanisation and the wider-area retries. The
answer names the place in the user's own words, and KMA gets that spot's
grid point. Eval: evals/test_weather_district_place.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:08:15 +09:00
javis-bot
1bef5db912 chore(app): version 0.1.28
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:57:23 +09:00
javis-bot
17927b6450 feat(weather): KMA forecasts for a later day (days_ahead, days)
getWeather takes days_ahead (days from today, up to 10) and days (a span,
2 for a weekend). Inside Korea with the app's KMA key: 단기예보 for today
to 3 days ahead (morning and afternoon weather, lowest and highest, highest
chance of rain), then 중기기온 and 중기육상 for the nearest of 32 KMA cities
and its region. Days KMA does not give, and every KMA failure, come from
open-meteo's daily forecast. days_ahead 0 with more days gives the weather
now, then each day. Eval: evals/test_weather_forecast_days.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:57:12 +09:00
javis-bot
ecde1e6bd0 chore(app): version 0.1.27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:05:45 +09:00
javis-bot
282f114e60 feat(weather): KMA current weather inside Korea with the app's stored key
The 기능 screen's 기상청 날씨 card stores the KMA API hub key encrypted
(kmaApiKey, masked in logs). On start the app passes it to the worker in
JARVIS_KMA_API_KEY; the worker takes it out of its environment before any
child process starts. Inside Korea getWeather reads 초단기실황 and the
coming hour's sky from 초단기예보, and falls back to open-meteo, then
met.no, on any KMA failure. The key is in KMA's URLs, so failures are
logged by their kind only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:05:31 +09:00
javis-bot
e99831e540 fix(gitignore): keep ignoring a leftover Gemini login folder
Removing the Gemini backend also dropped the rule, so an older checkout's
docker/gemini-oauth/oauth_creds.json could be committed by git add -A.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:30:38 +09:00
javis-bot
68d0333698 chore(app): version 0.1.26
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:24:11 +09:00
javis-bot
af1276cf6b refactor(search): remove the Gemini search backend
The user asked to drop it: nobody had the Gemini CLI login or a key set up,
and Exa then Wikipedia cover the voice use. webSearch is now live Chrome,
Exa, Brave (opt-in), Wikipedia. Removes the CLI/REST backends, the
GEMINI_* settings, the Docker login mount and checks, and updates the specs
and docs/llm_contexts.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:24:11 +09:00
javis-bot
59c3f33c2c test(notes): prepare the pick with the app's offer and require it to be reused
The partial-transcript eval now offers chat, schedule, notes and screen as
voice_worker/brain.py does, sets those backends for the reply, and fails
when the intent judge made no pick or the pick lacks personalNotes, so it
cannot pass by re-routing the final text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:30:49 +09:00
javis-bot
59a9277613 fix(weather): met.no sleet is sleet, not freezing rain
met.no's sleet symbols were mapped to WMO 66/67 (freezing rain), so the
answer said 어는 비. They now map to the WMO 4677 rain-and-snow codes
68/69 and 83/84 (진눈깨비, 진눈깨비 소나기).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:22:42 +09:00
javis-bot
d31d75f18f fix(planner): plan in the query's own writing system only
Live 2026-11-18 '자비 사과랑 배 중에 뭐가 더 달아' was searched as
'사과와 배 중哪个가 더 단가요'. The rule and the user message's closing line
now say to use only the query's letters.

Eval test_search_in_users_language.py, new case fruit (qwen3.6-35b-a3b,
game PC): Han in the search 7/7 before; rule only 3/3, closing line only
3/3, both 0/6; whole file 12/12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:22:17 +09:00
javis-bot
64dceb816e test(notes): eval the pick prepared from the partial transcript
The v2 brain reuses the intent judge's pick made while the person still
speaks. Eval: partial '자비스 앞으로 나한테는 짧게 대답' -> final request saves
the note. Old description 0/3, current 3/3 (qwen3.6-35b-a3b, game PC);
whole file 18/18.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:18:12 +09:00
javis-bot
2238ddbe42 feat(search): search with Exa, Gemini only with a login, title-guarded Wikipedia
DuckDuckGo now serves only its bot-check page (game PC and host alike), so
every search fell to Wikipedia full-text search, which picked unrelated
pages (박진영 for songs for a rainy day, 대한민국 for a graphics card
ranking). Live 2026-11-18 answers took 9-15 s and were wrong.

- Exa's free MCP endpoint (no key) is the first provider after the
  live-broadcast Chrome; it returns page text, so no fetch is needed.
- Gemini runs only when Exa found nothing and a CLI login (oauth) or key
  (apikey) is present; without a login the CLI is no longer started
  (about 2 s per search).
- Wikipedia is used only when the page title is named in the query;
  otherwise the honest nothing-found envelope.
- Letters of a script the user did not use are dropped from the query
  (the model searched '사과와 배 중哪个가 더 단가요').
- DuckDuckGo removed; spec, docs and tests follow the new order.
- weather: any forecast failure (not only requests errors) tries met.no.

Real model (qwen3.6-35b-a3b, game PC), real network: graphics card
ranking, apples vs pears, rainy-day genres and black holes all answered
correctly in 3.7-5.2 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:11:28 +09:00
javis-bot
680d4fbdc1 fix(weather): fall back to met.no when the open-meteo forecast does not answer
Live 2026-11-18 the open-meteo forecast timed out three times (geocoding
worked) and every weather question ended in 'service unavailable'. The
forecast now waits 6s instead of 10s and then asks met.no (no key),
mapping its symbol to the WMO code the Korean sentence uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 01:59:49 +09:00
javis-bot
c51e5b2f68 fix(notes): say 'save' in the first 120 characters so the planner plans the note
The planner and router read only the first 120 characters of a tool's
description. personalNotes started by describing the notes, so for
'앞으로 나한테는 짧게 대답해줘' the planner planned a plain reply and the
answer promised to be brief from now on without saving anything.

Eval (qwen3.6-35b-a3b, game PC): from-now-on cases 0/6 -> 6/6, the
one-off and note-shapes-answer cases still pass (15/15).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 01:50:16 +09:00
javis-bot
1c06b16bd4 chore(app): version 0.1.25
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
44387685af fix(app): log the card's real free memory, and short answers get their line too
Free memory is nvidia-smi's memory.free in the app and the voice worker, so the
driver's reserved share no longer counts as free. A card reading still running
when an answer ends finishes on its own and then logs, so an answer shorter
than one reading is logged too, and the answer still never waits for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
61b68130c6 chore(app): version 0.1.24
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:48 +09:00
javis-bot
f1600f09c2 feat(app): log the graphics card's free memory after the LLM starts and while each answer is written
The app logs how much of the card is free once the model answers, and the
voice worker reads the card once a second while an answer or a scheduled task
is written and logs the least free memory seen (no text, llm tab), so a slow
answer can be matched to a full card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:20 +09:00
javis-bot
ce03424ba4 docs(spec): PC control tool, next-message consent, agent-desktop in the app
pcControl (v2 M7): windows, read, click, type, keys, run, close through the
app's agent-desktop. Ask-level actions wait for the asker's own next message
and then cover that window's asked kinds for the reply; payments, passwords,
deleting files and admin rights are filtered before asking and never done.
The app refuses during an anti-cheat game and the emergency stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 14:04:43 +09:00
javis-bot
d02092662b chore(app): version 0.1.23
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:07:48 +09:00
javis-bot
4adc6a8d72 fix(app): resident launcher apps are not games, and the TTS keeps its share of the card
On the game PC, Wallpaper Engine (steamapps\common\wallpaper_engine) and the
Riot client's crash handler (Riot Games\Riot Client) were taken for games
running behind other windows as the services started, so no reading of other
programs was kept and the game stage placed 0 GiB of experts instead of 2.

- A launcher's own folders and non-game apps under its game folder are not
  game folders (Steamworks Shared, wallpaper_engine; Launcher, Epic Online
  Services; Riot Client).
- The game log tab names the program found running at the start.
- The placement keeps the TTS's share (MeloTTS, 1005 MiB, M0 tts_melo) beside
  the STT's, and takes the worker's loaded models off what the card holds, so
  a start (TTS not yet loaded) and a switch (TTS loaded) place the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:07:45 +09:00
javis-bot
4993942040 chore(app): version 0.1.22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 10:11:59 +09:00
javis-bot
ff78c0ac41 feat(app): the LLM always takes one request at a time, and only the latest call is answered by default
A chat channel answering every message no longer starts Strata with
--parallel 2: on the 16 GB card that read prompts about three times
slower (2026-11-16, answers took 10-20 s). Chat and voice answers now
take turns at Strata. response.policy defaults to latest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 10:01:37 +09:00
javis-bot
8c61d7a821 perf(llm): short one-shot calls say so to Strata
Every call that no later request continues (call_llm_direct,
call_llm_streaming, the warm-up and the v1 intent judge) now sends
"strata_checkpoint": false on the OpenAI-compatible path. Strata's
Qwen3.6 engine then gives that call's session away first, so with two or
more requests at once a reply's tool turn keeps its session through the
short calls made between its steps. The reply loop's chat_with_messages
is not marked (its next step continues it); the Ollama API gets no new
field. With one request at once nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 07:06:25 +09:00
javis-bot
07615ee7a1 chore(app): version 0.1.21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 21:13:39 +09:00
javis-bot
4d0ae91e8a perf(reply): v2 tool turns get the tool guidance in short, and a shorter reply-language default
Strata reads every reply prompt from the start in 2,048-token steps. The v2
compact prompt carried v1's full tool guidance and constraints (about 900
Qwen3.6 tokens) on every turn with a data tool, and the reply-language
default (about 120 tokens) twice.

COMPACT_TOOL_GUIDANCE keeps the same rules (act instead of refusing, facts
only from the result, look up named things, auto-derived and self-contained
arguments) in about a quarter of the tokens, without English examples; the
reply-language default says the same in half. v1's prompts are unchanged.

Measured on Strata (Qwen3.6, 20 lines of recent conversation): a weather
turn's prompt 3,436 -> 2,722 tokens, a search turn's second call
4,096 -> 3,383. Evals (compact prompt + intent judge, Qwen3.6): 24/24
before, 30/30 twice after (weather location, follow-up arguments, tool-skip
confabulation, search language, reply language, Possessor fidelity, reply
length, multi-turn context).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 21:13:08 +09:00
javis-bot
7da5bca783 fix(reply): a one-step turn whose model skips the tool twice is planned and guarded
After the one nudge, a second answer without the tool no longer goes out
as it is: the turn takes the planned path (the planner runs then, one
extra call only on this failure) and the skipped-tool guard runs the
planned tool, so a made-up "24 degrees" answer cannot reach the user. A
plan that needs no tool (a question back to the user) still lets the
answer go out.

Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs):
the five tool-skip/weather/screen/follow-up/router files 72/84 (71/84 on
0.1.20, same known failures); Possessor director-then-filmography 0/3,
also 0/3 on 0.1.20 (qwen3:8b stops after one search; webSearch turns are
still planned).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:09:11 +09:00
javis-bot
a6932d9b16 perf(reply): v2 turns with one one-step tool skip the planner
When the router's only pick is a tool that does its job in one call
(getWeather, logMeal, fetchMeals, screenView) and the chat model calls
tools natively, the planner no longer runs: on Strata it averaged 2.5 s
per tool turn (2026-11-16 16:45-16:58 engine log, 9 calls) before the
answer could start. Search, browsers, schedule, notes, chat channels and
picks of two or more tools still plan. The rule reads the router's pick,
not the words, so it is language-agnostic.

Without a plan the skipped-tool guard has nothing to run, so these turns
get one nudge instead: an answer given before any tool ran is dropped and
the model is asked to call the tool (or ask for a missing detail).

Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs of
tool_skip_confabulation, weather_autoderive_location, screen_view_tool,
followup_supplies_missing_tool_arg, tool_router_implicit): 71/84 before,
72/84 after, same known failures; planner calls 14 -> 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:00:36 +09:00
javis-bot
6a9b66bdcd chore(app): version 0.1.20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:26:56 +09:00
javis-bot
cda8db7fce fix: play on YouTube in one step, Korea as the default location, searches in the user's language
- botBrowser `play`: searches YouTube for the words as said, opens the first video (not an ad,
  without the playlist) and starts it, in one call (an address given opens that page); `type`
  also takes `query`. Live 0.1.19: the model stopped on YouTube's home page or the results, a
  retry was refused as a third call, and the answer still said the video played.
- `brain.location` (기본 위치, default 대한민국) goes to the brain as `location_name`: the time line
  says it without GeoIP (no database on the game PC), and getWeather without a place uses it.
- Planner and webSearch keep search words in the user's language (never translated into
  English), so recommendations fit where the user is.

Evals (qwen3:8b, 3 runs): play ending on a playing video found with the words as said 0/6 -> 5/6;
Korean searches for Korean requests 6/9 -> 8/9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:26:30 +09:00
javis-bot
65294f3f97 chore(app): version 0.1.19
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:12:20 +09:00
javis-bot
420c9c73f3 fix: stopped sentences recorded only as far as heard, search words as plain query, paused video said so
- Playback sends how long the clip playing when a request was stopped had played (tts_played
  last_ms); the worker cuts the last sentence of that recording to it.
- botBrowser open takes the search words as plain `query`: the tool puts them, encoded, into the
  address's empty search parameter or types them into the site's search box. The description
  tells the model to search this way and never put words into an address itself (live: 아이유
  hand-encoded into 아이우). Eval (qwen3:8b, 3 runs): right words reach YouTube 1/3 -> 3/3.
- When a page's main video does not play (also after the tool pressed play, or it failed to load),
  a note after the page says it is not playing and not to say it plays. Eval: a paused video
  reported as playing 1/3 -> 0/3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:12:04 +09:00
javis-bot
9a91bc9ab6 chore(app): version 0.1.18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:56:40 +09:00
javis-bot
ec2672cd4a fix: recordings keep only what was played, newest waiting request kept, video pages opened play
- Playback tells the worker how much of each request was played (tts_played); the worker keeps a
  spoken answer's recording only that far (an answer stopped mid-way no longer keeps its unheard
  rest, and is marked stopped); nothing played or never reported: no file.
- Old recording day folders are also removed with the first recording of each new day.
- response.interrupt off, latest: a request whose final comes after a newer one's is replaced
  instead of replacing it.
- botBrowser presses play on a page's paused main player also after open, not only after a click.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:56:40 +09:00
javis-bot
fdcaae4820 chore(app): version 0.1.17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:43:10 +09:00
javis-bot
72518ca50e feat(app): addressed calls follow who is in the voice channel
Alone with Jarvis, whatever the speaker says is answered without asking the judge. With others
present the judge is told how many people are in the channel and how many may call Jarvis: with
one other person most of what the speaker says counts as said to Jarvis (unclear is yes); with
more, or not known, the question is the group one it was before. Live 2026-11-16: alone in the
channel, questions without the wake word got no answer because the judge assumed a group.
Eval (qwen3:8b, pair cases): 3/7 before, 6/7 after; group cases unchanged.

Also: the protocol test fixture's brain takes the trace argument (the test failed since traces).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:43:10 +09:00
javis-bot
80ce9281a3 chore(app): version 0.1.16
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:29:51 +09:00
javis-bot
cbe898d3d0 feat(app): log screen saves (shown lines, conversation, voice recordings), stop-or-finish on a new request, video pages that play
- 로그 screen: "보이는 글 txt로 저장" (logs_save_text), a 대화 view that saves the days'
  conversation with every trace in full (history_export), and a 목소리 view with the
  voice.record switch and the days' recordings as a zip with an index (voice_export).
- Voice worker: with JARVIS_RECORDINGS_DIR (voice.record, off by default) each final and each
  spoken answer is kept as a WAV with an index line; day folders older than 7 days removed.
- response.interrupt (default on): off, latest/important finish the running answer first.
- Bot browser: Chrome lets pages start video and sound; botBrowser reports each page's media
  state, waits for a player to start after open/click, and presses play on a video page a click
  led to when its player stayed paused (live: a clicked YouTube video stayed paused).
- webSearch reads `query` when `search_query` is missing (live: both searches of an answer failed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:27:57 +09:00
javis-bot
1cdfbf5151 chore(app): version 0.1.15
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:33:38 +09:00
javis-bot
b2b6f92943 fix(app): 대화 shows a trace before a quick answer ends
The screen re-read its window every 3 seconds unless a trace was already
shown as being made, so an answer that ended between two reads came with
its trace. It now asks twice a second for history_stamp (the newest
record and trace ids and the steps of the traces being made, cheap and not
logged) and reads the window again as soon as it changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:33:38 +09:00
javis-bot
1f2a8ec05e chore(app): version 0.1.14
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:20:33 +09:00
javis-bot
04b516dbc1 feat(app): 한 일 as a view of the 로그 screen, shortcuts to a left-behind copy start the current Jarvis
The 한 일 screen is gone: 로그 has 로그 and 한 일 views switched at the top.

An update before 0.1.12 could install into Program Files while the desktop
shortcut kept pointing at the first install's folder, an old copy that no
update reaches: its jarvis.exe shows the old icon and starts a second, old
Jarvis next to the running one. The installer now points every desktop and
start menu shortcut to another Jarvis install's jarvis.exe at this install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:20:33 +09:00
javis-bot
b386c905b2 chore(app): version 0.1.13
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:04:00 +09:00
javis-bot
b29150a767 feat(app): updates on the 앱 screen, traces that fill in live on 대화
The separate 업데이트 screen is gone: its card is first on the 앱 screen
and a newer release renames the 앱 button. On 대화 a trace appears when
the question is recognised, opens and fills in step by step (re-read
every second while one is being made) and stays open with its answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:03:45 +09:00
javis-bot
6fc3ad3d57 feat(worker): answer traces stored from the question on and filled in as the answer is made
A trace is now kept the moment the request arrives (at the time it was
asked, ms -1), its steps are stored as the engine adds each one, and it
ends with the answer. Traces a killed run left unfinished are closed at
start. The 대화 screen reads a trace still being made as pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:03:45 +09:00
javis-bot
fd7ccab55b fix(app): shortcut icons in start menu folders and any install folder, updates go where Jarvis is
- The icon pass also walks the start menus' folders, so a shortcut kept
  in one (and a later icon change) is not missed.
- The install folder and icon path reach PowerShell through the
  environment: a folder name with an apostrophe broke the whole command.
  A failure is now a line in the install details.
- InstallDirRegKey read the 32-bit registry view before .onInit switched
  to the 64-bit one where the folder is written, so a silent update of
  Jarvis installed elsewhere put a second copy in Program Files. .onInit
  now reads the 64-bit key; a folder given with /D wins.
- Version 0.1.12.

Checked on the Windows VM with Jarvis 0.1.6 in C:\Tools\Kim's Jarvis and
shortcuts on both desktops and in start menu subfolders: 0.1.11 left the
desktop and subfolder shortcuts old; 0.1.12 (no /D) updated that folder
and every shortcut, and a fresh 0.1.12 still goes to Program Files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:56:24 +09:00
javis-bot
2b832c1c3b feat(app): 앱 last in the side menu, version 0.1.11
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:47:03 +09:00
219 changed files with 25409 additions and 2153 deletions

View File

@@ -122,7 +122,7 @@ CHROME_START_URL=about:blank
# Screen-share + browser mode.
# true = the bot may go Live (screen-share the VNC desktop) and drive the
# on-screen browser for real-time info (search / play / read screen).
# false = no screen share; voice only, real-time info via the Gemini API.
# false = no screen share; voice only, real-time info via the normal web search.
STREAM_BROWSER=true
# Optional: profile dir for browser-based Google search in plain text turns
# (no active broadcast). When set, the search helper opens Chrome against this
@@ -133,27 +133,6 @@ STREAM_BROWSER=true
# Google does not challenge it). Use a DEDICATED dir, not your everyday Chrome
# profile, to avoid the "profile in use" lock while Chrome is open.
CHROME_USER_DATA_DIR=
# Gemini auth for real-time info when STREAM_BROWSER=false.
# oauth = use the Gemini CLI with a Google-account login (no API key).
# Install once: npm i -g @google/gemini-cli ; then run `gemini` and
# "Sign in with Google". Uses the CLI's built-in web-search grounding.
# NOTE (2026-06): Google is blocking personal Google accounts on this
# path ("This client is no longer supported for Gemini Code Assist for
# individuals"). Workspace/org accounts may still work; personal
# accounts should use apikey below instead.
# apikey = REST path; needs GEMINI_API_KEY below
# (get one at https://aistudio.google.com/app/apikey). Recommended for
# personal Google accounts now that individual OAuth login is blocked.
# Either way, real-time search fail-opens to DDG/Brave/Wikipedia if Gemini is
# unavailable, so this is optional, not required.
GEMINI_AUTH=oauth
GEMINI_API_KEY=
GEMINI_MODEL=gemini-2.0-flash
# OAuth login source for Docker. The container mounts this into ~/.gemini.
# Default (blank) = ./docker/gemini-oauth (project-local, cross-platform). Seed
# it once: cp -r ~/.gemini/. docker/gemini-oauth/ (copy the whole login state).
# Or point at an existing host login instead, e.g. GEMINI_OAUTH_DIR=~/.gemini
GEMINI_OAUTH_DIR=
# ---------------------------------------------------------------------------
# VNC screen broadcast

7
.gitignore vendored
View File

@@ -29,7 +29,6 @@ src/jarvis/_version.py
.env.bak*
*.bak
# Gemini CLI OAuth login (account tokens) seeded for GEMINI_AUTH=oauth in Docker.
# Keep the dir (.gitkeep) but never commit the login files.
docker/gemini-oauth/*
!docker/gemini-oauth/.gitkeep
# Gemini CLI OAuth login (account tokens) from the removed Gemini search backend. An
# older checkout may still hold docker/gemini-oauth/oauth_creds.json; never commit it.
docker/gemini-oauth/

View File

@@ -21,12 +21,13 @@ Any code change must either adhere to our spec files perfectly or you should ask
| `src/jarvis/tools/builtin/tool_search.spec.md` | toolSearchTool escape hatch for mid-loop tool routing | Re-runs the same router; never removes stop/self; capped per reply |
| `src/jarvis/tools/external/mcp_runtime.spec.md` | Persistent MCP runtime: per-server long-lived stdio session, queue-based dispatch, retry on transient session loss | One worker per server keyed by config; calls to the same server serialise; `MCPServerSessionError` for session-level failures; opt-in `idle_timeout_sec` for stateless servers |
| `src/jarvis/reply/prompts/prompts.spec.md` | System/user prompt templates | — |
| `src/jarvis/tools/builtin/web_search.spec.md` | webSearch tool: STREAM_BROWSER routing (browser/Gemini), cascade fetch, SSRF guard, prompt-injection fence, links-only envelope | Untrusted web content is fenced as data, not instructions; rank preference over speed; honest failure over confabulation |
| `src/jarvis/tools/builtin/web_search.spec.md` | webSearch tool: live-broadcast Chrome, then Exa (keyless), Brave (opt-in), title-guarded Wikipedia; foreign-script drop, SSRF guard, prompt-injection fence, honest envelopes | Untrusted web content is fenced as data, not instructions; rank preference over speed; honest failure over confabulation |
| `src/jarvis/tools/builtin/chat_channels.spec.md` | chatChannels tool (v2): list, read, search, send, edit, react, delete, pin registered Discord chat channels through the voice worker and app | Offered only with a chat backend; per-channel switches; chat is fenced data; delete needs the user's yes in a later turn |
| `src/jarvis/tools/builtin/schedule.spec.md` | schedule tool (v2): reminders, timers and scheduled tasks of the asker; the app keeps, fires and delivers them | Offered only with a schedule backend; only the asker's own; tasks get read-only tools; never claimed without the tool |
| `src/jarvis/tools/builtin/personal_notes.spec.md` | personalNotes tool (v2): each person's standing instructions for how they are answered; the app keeps them and sends the asker's with every reply | Offered only with a notes backend; only the asker's own, only in their answers; fenced, never permits an action |
| `src/jarvis/tools/builtin/collected_news.spec.md` | collectedNews tool (v2): read the background collection's results (targets, newest items) | Offered only with the app's collection file and the feature on; web content fenced as data; read-only, no LLM |
| `src/jarvis/tools/builtin/screen_view.spec.md` | screenView tool (v2): see the whole screen, one monitor or one window through the app; pictures go to the reply model for that reply only | Offered only when the app makes screen viewing available; the app decides who may see; screen text is data; pictures never kept or logged |
| `src/jarvis/tools/builtin/pc_control.spec.md` | pcControl tool (v2 M7): list windows, read a window, click, type, keys, run and close programs through the app's agent-desktop | Offered only when the app makes PC control available; ask-level actions need the asker's own next message; payments, passwords, deleting files and admin never; screen text is data |
| `src/jarvis/tools/builtin/bot_browser.spec.md` | botBrowser tool (v2): open, read, click, type, back in the bot's own Chrome over CDP | Offered only when the app started the bot browser; only http(s); page content is fenced data; no LLM call |
| `src/jarvis/tools/builtin/browse_and_play.spec.md` | browseAndPlay tool: play YouTube on the shared screen (screen-share mode only) | Node layer owns Chrome/CDP; mode-gated; fail-open, no LLM call |
| `src/jarvis/tools/builtin/nutrition/log_meal.spec.md` | logMeal tool: single-property schema for planner fast-path, internal nutrition extraction, untrusted-data fence, follow-ups | Public schema is a single optional `meal` string; nutrition fields are internal; user text is fenced as data |

View File

@@ -39,16 +39,11 @@ RUN wget -q -O /tmp/chrome.deb https://dl.google.com/linux/direct/google-chrome-
# --- bun (Discord bot runtime/package manager) ---
RUN curl -fsSL https://bun.sh/install | bash
# --- Node.js + Gemini CLI (the GEMINI_AUTH=oauth real-time search path shells
# out to `gemini`; it needs a real Node runtime, not bun). The account
# login itself is provided at runtime via the mounted ~/.gemini (see
# docker-compose.yml); here we only install the binary. ---
ARG GEMINI_CLI_VERSION=0.46.0
# --- Node.js (the on-screen browser search helper, browse-search.mjs, needs a
# real Node runtime, not bun) ---
RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& npm install -g "@google/gemini-cli@${GEMINI_CLI_VERSION}" \
&& rm -rf /var/lib/apt/lists/* \
&& gemini --version
&& rm -rf /var/lib/apt/lists/*
# --- Python brain/bridge deps (slim set) ---
COPY bridge/requirements-bridge.txt /app/bridge/requirements-bridge.txt

View File

@@ -36,9 +36,11 @@
- 원하면 게임 중에는 봇을 일시정지할 수 있습니다.
- **안티치트 보호**: 뱅가드 같은 안티치트가 걸린 게임이 실행 중이면 키보드·마우스 자동 조작을 막고 그 이유를 알려 줍니다.
- **기존 기능 유지**: 메모리, 날씨, 웹 검색, 식단 기록, MCP, 브라우저 제어를 그대로 가져갑니다.
- **기상청 날씨**: 관리 앱 "기능" 화면에 기상청 API허브 인증키를 넣으면 한국 안의 현재 날씨와 10일 뒤까지의 예보("내일 날씨", "주말 날씨")를 기상청에서 받습니다. "부산 연제구"처럼 구 단위로 물으면 그 구의 날씨를 답하고(장소는 OpenStreetMap에서 찾습니다), "자세히" 물으면 습도, 바람, 강수량까지 답합니다(초단기실황조회, 초단기예보조회, 단기예보조회, 중기기온조회, 중기육상예보조회 활용신청 필요). 키가 없거나 한국 밖이면 open-meteo에서 받고, 현재 날씨는 그다음 met.no에서 받습니다.
- **관리 앱**:
- 첫 실행 마법사, 대시보드, 모델 다운로드, 기능별 켜기/끄기를 제공합니다.
- 로그 화면에서 로그를 보고, 복사하고, 내려받을 수 있습니다. 비밀값은 자동으로 가립니다.
- 로그 화면의 "대화"에서 날짜별 대화 내용(사람 말, 자비스 생각과 도구 호출, 답)을 txt로, "목소리"에서 녹음한 목소리를 zip으로 저장합니다. 목소리 녹음은 기본으로 꺼져 있고, 켜면 이 PC에만 7일 동안 남습니다.
## 요구 사항 (예정)

4
app/Cargo.lock generated
View File

@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.10"
version = "0.1.31"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -3324,8 +3324,6 @@ dependencies = [
[[package]]
name = "songbird"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "371362adce10df3244fbf93475ac53391aedd425482f160a9dbb3e042577de8b"
dependencies = [
"aead",
"aes-gcm",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-app"
version = "0.1.10"
version = "0.1.31"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"
@@ -44,6 +44,10 @@ jpeg-decoder = "0.3"
name = "vdisplay_elevated"
harness = false
# songbird 0.6.1 with the Jarvis patch for the first packets of a speaker (vendor/songbird/JARVIS-PATCH.md).
[patch.crates-io]
songbird = { path = "vendor/songbird" }
[profile.release]
lto = "thin"
codegen-units = 4

6
app/shell/Cargo.lock generated
View File

@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.10"
version = "0.1.31"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -2666,7 +2666,7 @@ dependencies = [
[[package]]
name = "jarvis-shell"
version = "0.1.10"
version = "0.1.31"
dependencies = [
"jarvis-app",
"serde_json",
@@ -5562,8 +5562,6 @@ dependencies = [
[[package]]
name = "songbird"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "371362adce10df3244fbf93475ac53391aedd425482f160a9dbb3e042577de8b"
dependencies = [
"aead",
"aes-gcm",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-shell"
version = "0.1.10"
version = "0.1.31"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"
@@ -21,6 +21,10 @@ tauri-plugin-single-instance = "2"
serde_json = "1"
tokio = { version = "1", features = ["full"] }
# songbird 0.6.1 with the Jarvis patch for the first packets of a speaker (vendor/songbird/JARVIS-PATCH.md).
[patch.crates-io]
songbird = { path = "../vendor/songbird" }
[profile.release]
lto = "thin"
codegen-units = 4

View File

@@ -19,7 +19,6 @@ SetCompressor /SOLID lzma
Name "Jarvis"
OutFile "${OUT}"
InstallDir "$PROGRAMFILES64\Jarvis"
InstallDirRegKey HKLM "Software\JarvisBot" "InstallDir"
RequestExecutionLevel admin
BrandingText "Jarvis ${VERSION}"
VIProductVersion "${VERSION}.0"
@@ -149,6 +148,13 @@ Function .onInit
Abort
${EndIf}
SetRegView 64
; An update goes where Jarvis is installed (the key is in the 64-bit view, which InstallDirRegKey
; would not read); a folder given with /D wins.
ReadRegStr $0 HKLM "Software\JarvisBot" "InstallDir"
${If} $0 != ""
${AndIf} $INSTDIR == "$PROGRAMFILES64\Jarvis"
StrCpy $INSTDIR $0
${EndIf}
; Shortcuts for every user of the PC, like the Program Files install itself.
SetShellVarContext all
!insertmacro TakeLock
@@ -248,13 +254,24 @@ Section /o "바탕 화면 바로 가기" SecDesktop
CreateShortCut "$DESKTOP\Jarvis.lnk" "$INSTDIR\jarvis.exe" "" "$INSTDIR\${ICON_FILE}" 0
SectionEnd
; Every shortcut to jarvis.exe on the desktops and in the start menus (the person's own too, and the
; installer's from an earlier version, which an update does not make again) shows this version's
; icon file; then Explorer redraws its icons. The 64-bit PowerShell (SysNative from this 32-bit
; installer): a 32-bit one reads a link to Program Files as one to Program Files (x86).
; Every shortcut to jarvis.exe on the desktops and in the start menus, their folders included (the
; person's own too, and the installer's from an earlier version, which an update does not make
; again) shows this version's icon file; then Explorer redraws its icons. A shortcut to another
; Jarvis install's jarvis.exe (a folder with uninstall.exe and python-src, left behind when an
; earlier update installed into Program Files instead) starts this one instead, so it opens the
; Jarvis that runs, not an old copy. The paths go in through
; the environment, so no install folder name can break the command. The 64-bit PowerShell
; (SysNative from this 32-bit installer): a 32-bit one reads a link to Program Files as one to
; Program Files (x86).
Section "-Icons"
nsExec::Exec `"$WINDIR\SysNative\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -NonInteractive -Command "$$w = New-Object -ComObject WScript.Shell; foreach ($$d in 'Desktop', 'CommonDesktopDirectory', 'Programs', 'CommonPrograms') { Get-ChildItem -LiteralPath ([Environment]::GetFolderPath($$d)) -Filter *.lnk -EA 0 | ForEach-Object { $$l = $$w.CreateShortcut($$_.FullName); if ($$l.TargetPath -ieq '$INSTDIR\jarvis.exe') { $$l.IconLocation = '$INSTDIR\${ICON_FILE},0'; $$l.Save() } } }"`
System::Call 'kernel32::SetEnvironmentVariable(t "JARVIS_EXE", t "$INSTDIR\jarvis.exe")'
System::Call 'kernel32::SetEnvironmentVariable(t "JARVIS_ICON", t "$INSTDIR\${ICON_FILE},0")'
System::Call 'kernel32::SetEnvironmentVariable(t "JARVIS_DIR", t "$INSTDIR")'
nsExec::Exec `"$WINDIR\SysNative\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -NonInteractive -Command "$$w = New-Object -ComObject WScript.Shell; foreach ($$d in 'Desktop', 'CommonDesktopDirectory', 'Programs', 'CommonPrograms') { Get-ChildItem -LiteralPath ([Environment]::GetFolderPath($$d)) -Filter *.lnk -Recurse:($$d -like '*Programs') -EA 0 | ForEach-Object { $$l = $$w.CreateShortcut($$_.FullName); $$t = $$l.TargetPath; if ($$t -and (Split-Path -Leaf $$t) -ieq 'jarvis.exe' -and (Test-Path -LiteralPath (Join-Path (Split-Path $$t) 'uninstall.exe')) -and (Test-Path -LiteralPath (Join-Path (Split-Path $$t) 'python-src'))) { if ($$t -ine $$env:JARVIS_EXE) { $$l.TargetPath = $$env:JARVIS_EXE; $$l.WorkingDirectory = $$env:JARVIS_DIR }; $$l.IconLocation = $$env:JARVIS_ICON; $$l.Save() } } }"`
Pop $0
${If} $0 != 0
DetailPrint "바로 가기 아이콘을 바꾸지 못했습니다 ($0)"
${EndIf}
System::Call 'shell32::SHChangeNotify(i 0x08000000, i 0, p 0, p 0)' ; SHCNE_ASSOCCHANGED
SectionEnd

View File

@@ -81,6 +81,13 @@ manages the MCP servers it connects.
listed. On start the app passes the enabled servers' stored values to the worker in its
environment, `JARVIS_MCP_ENV` (`{server: {NAME: value}}`); the brain takes that variable out of
its own environment and gives each server only its own values.
- **The KMA key** (기상청 API허브, for `getWeather` inside Korea): the 기능 screen's "기상청 날씨" card
takes it (`kma_key_set`; an empty value removes it) and shows it only masked. It is stored as
`kmaApiKey`, never in the settings file or `config/brain.json`, and saving `mcpServers` keeps it.
On start the app passes it to the worker in `JARVIS_KMA_API_KEY`; the worker takes that variable
out of its environment before any child process starts, and only the weather tool uses it
(the weather now and the forecast up to 10 days ahead inside Korea from KMA, else open-meteo;
the weather now then met.no). Without a key nothing changes.
- Each server's outcome (its tool count, or why it failed) is a worker line in the LLM log tab,
also without debug logging. MCP servers are programs the PC runs; the screen says to add only
trusted ones.
@@ -204,6 +211,53 @@ one window; the app captures and decides who may.
a title), who asked and whether it worked ("Activity").
- Logs carry the action, target kind, size and time taken, never a title or a picture.
## PC control (`app/src/pc.rs`)
`docs/v2/pc-features.md` ("키보드·마우스 제어와 안티치트"). The brain's `pcControl` tool
(`src/jarvis/tools/builtin/pc_control.spec.md`) operates programs; the app runs every action through
agent-desktop and decides who may.
- **agent-desktop**: version 1.2.0 (Apache-2.0), pinned with its SHA-256, shipped in the installer
next to `jarvis.exe` (`agent-desktop.exe`, about 4 MB) and checked before every start of the
services (a wrong hash means not available). One CLI run per command (M0 `desktop`: 0.10 s, as
fast as its MCP server), its APPDATA and LOCALAPPDATA in the data folder (`runtime/agent-desktop`),
a 15 s limit per command. Actions are headless (no cursor, no focus), except `keys`, which brings
the window to the front first. `--force` (which lifts its block on Alt+F4, Win+R, Ctrl+Shift+Esc
and the like) and the raw mouse and key commands are never used.
- **Available** only on Windows with `features.pcControl` on, someone in `pcControl.allowedUsers`
and agent-desktop present; otherwise `pcControl` is in the brain's `disabled_tools`. Settings
apply on the next start.
- **Who and what**: every `pc_action` (`voice_worker.spec.md`, "PC control") is decided for its
asker by `permissions::decide` with the action's kind (`windows`, `read`: View; `click`, `close`:
Click; `type`, `keys`: Type; `run`: Run). Refused: people not registered ("PC 제어 화면에 등록된
사람만 PC를 조작할 수 있습니다"), a kind at `never` ("PC 제어 화면에서 이 동작이 꺼져 있습니다").
At `ask` the action is done only with `consent: true`, else the answer is `ask: true` with the
kinds and the window (the brain asks the person; the app does not keep consent).
- **Never**: typing into a password field (agent-desktop reports it, or Windows UI Automation's
IsPassword) is refused ("비밀번호 칸에는 입력하지 않습니다"); a window run as administrator is
refused before acting ("관리자 권한으로 실행된 창은 조작할 수 없습니다"), and `run` never asks for
administrator rights.
- **Anti-cheat**: while a game with an anti-cheat runs, every action but `windows` and `read` is
refused with "<안티치트>가 적용된 게임(<게임>)이 실행 중이어서 키보드/마우스 제어를 할 수
없습니다". Found by its processes, checked at each action: Vanguard by Riot's game processes
(VALORANT, League of Legends; the driver is always loaded), Easy Anti-Cheat, BattlEye, FACEIT,
nProtect GameGuard and XIGNCODE3 by their own processes (list in `pc.rs`, grown by updates), and
`pcControl.antiCheatGames` (program file names the person added on the PC 제어 screen; no switch
turns the check off). The game's name is the game detection's when it runs, else the program's.
- **Emergency stop**: a global hotkey (`pcControl.stopHotkey`, default `Ctrl+Shift+F12`) stops at
once: the running agent-desktop command is killed and every action but `windows` and `read` is
refused ("비상 정지 중입니다. 단축키를 한 번 더 누르거나 PC 제어 화면에서 풀어 주세요") until the
hotkey is pressed again or the PC 제어 screen lifts it. The dashboard shows "PC 제어: 비상 정지";
an app restart lifts it.
- **Results**: `read` and every action answer with the window `{id, program, title}` and its
interactive elements read afresh after the action (at most 200), so the brain says it was done
only from what it sees. agent-desktop's error codes come back as Korean reasons (`STALE_REF`: "화면이
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting).
- **Activity**: each action but `windows` and `read` is `pc.<action>` with the program's file name
(never a title, element name or text), who asked and whether it worked ("Activity").
- Logs carry the action, the result code and the time taken; never a title, element name, value or
typed text.
## Personal notes (`app/src/notes.rs`)
`docs/v2/proactive.md` (제안 4). Each person's standing instructions for how Jarvis answers them,
@@ -229,7 +283,8 @@ records and "Traces"), so it works while the services are stopped. Nothing it re
- Each item is `said` (a voice transcript, or a chat message by someone other than Jarvis),
`answered` (a spoken answer, or Jarvis's own chat message; to whom, when it answers someone) or
`thought` (a trace: the question, the answer or none, how long it took and its steps). Jarvis's
`thought` (a trace: the question, the answer or none, how long it took and its steps so far, at
the time it was asked; `pending` while the answer is being made, with `ms` 0 until it ends). Jarvis's
chat messages are known by the bot's user id (the first part of the stored token is that id in
base64) and by the author of any chat answer to someone.
- `history_query` `{said, answered, thought, via, person, channel, find, since, until, session, before, from, limit}`: the
@@ -245,10 +300,13 @@ records and "Traces"), so it works while the services are stopped. Nothing it re
were left out, the people who spoke (their latest name) and the channels with records (with the
name given a chat channel in the settings), and `session`, when the latest start began (Unix
seconds, null before the first). Without the file: no items and `missing: true`.
- `history_stamp`: `{stamp}`, a string that changes when a record or trace is added and as a trace
still being made gets a step or ends (the newest ids, and the number and length of the steps of
the traces being made), cheap enough to ask twice a second; not logged.
## Activity (`app/src/activity.rs`)
`docs/v2/proactive.md` (제안 5). What Jarvis did outside its answers, for the 한 일 screen: each chat
`docs/v2/proactive.md` (제안 5). What Jarvis did outside its answers, for the 로그 screen's 한 일 view: each chat
channel action the brain's tool asked for (`chat.send`, `chat.edit`, `chat.react`, `chat.delete`,
`chat.pin`, `chat.unpin`), each schedule delivery (`schedule.remind`, `schedule.task`) and each
news notification (`collect.notify`), each screen capture (`screen.view`, "Screen viewing"), done or failed. The other PC control actions join it with M7. Ordinary answers are not recorded.
@@ -291,7 +349,7 @@ channel, titles and links only (no LLM, nothing Jarvis writes about them).
## Secrets (`app/src/secrets.rs`)
`config/secrets.dat` holds a JSON object of named secrets (`discordToken`, the MCP servers'
environment values `mcp:<server>:<NAME>`). On Windows the bytes are
environment values `mcp:<server>:<NAME>`, the KMA API hub key `kmaApiKey`). On Windows the bytes are
encrypted with DPAPI for the current Windows user (`CryptProtectData`, no prompt), so the file is
useless copied to another account or PC. Elsewhere (development only) the file is plain JSON with
owner-only permissions (0600). Writes are atomic. `get`, `set`, `remove`, `names`; `masked(value)` shows only
@@ -324,11 +382,26 @@ STT, LLM and TTS lines go to `stt`, `llm`, `tts` by their worker category), `bro
sentence dots around it), becomes `[비밀값]`. So a secret replaced or removed later never shows on
the screen, in a copy, a file or an export, also after a restart. The values masked are the
stored secrets and the ones the running services were started with (the bot token, the MCP
servers' values; `set_secrets_in_use` on each start): a key replaced or removed while the
servers' values, the KMA key; `set_secrets_in_use` on each start): a key replaced or removed while the
services run stays masked in their late lines (the worker and MCP servers keep it until they are
restarted), until the next start gives them only the current values.
- Errors the voice worker reports to the app (`error` messages: code and request id) are always
logged, debug logging or not, so the log screen and the error report show them.
- Saves the user asks for on the 로그 screen, each a file in `logs/` whose path comes back (kept
until the user removes it, like the zip; the 14-day cleanup takes only `<tab>-<day>.log`):
- `logs_save_text {text}`: the lines the 로그 view shows, masked again, as
`jarvis-log-view-<YYYYMMDD-HHMMSS>.txt`.
- `history_export {from, to}`: every item of the 대화 screen on those local days, in time order,
as `jarvis-talk-<from>-<to>.txt` (`history_view::export_text`): each record (time, who with
their ID, voice or chat and channel, the text), each answer (to whom), and each trace in full:
the question, the records and notes it was given (every line), the tools offered, the plan,
the model's thinking, every tool call (arguments, success, failure or skipped and why, the
whole result) and the answer. Conversation text, so never a log line; masked like an export.
- `voice_export {from, to}`: the recordings of those days (`recordings/<day>/`,
`voice_worker.spec.md`, "Recordings") as `jarvis-voice-<from>-<to>.zip`: the WAV files under
`<day>/`, each day's `index.jsonl`, and `index.txt` listing every file in time order with its
length, who said it and the transcript (or Jarvis's spoken text, and whether it was stopped
partway). No recordings: a zip whose `index.txt` says so.
## Service control (`app/src/control.rs`)
@@ -437,6 +510,9 @@ background (headless), or with `features.browserShare` in a window on the virtua
(`--remote-debugging-port=0`; the port is read from the profile's `DevToolsActivePort`, which is
removed before each start, and `/json/version` must answer within 30 s). The worker gets
`JARVIS_BROWSER_CDP` = `http://127.0.0.1:<port>`.
- Video and sound start when a page starts them (`--autoplay-policy=no-user-gesture-required`): the
botBrowser tool clicks with page scripts, which Chrome does not count as a user's gesture, so a
video page opened for "play X" stayed paused (live, 2026-11-16).
- **Egress proxy**: every Chrome connection goes through the app's proxy on a free loopback port
(`--proxy-server`, and `--proxy-bypass-list=<-loopback>` so not even loopback is direct; QUIC off,
WebRTC only through the proxy). The proxy takes `CONNECT host:port` (HTTPS, WebSocket) and plain
@@ -560,8 +636,8 @@ between the normal placement and the game placement as games come and go (not wi
engine's CPU threads (read from its log at the normal start, else a quarter of the logical CPUs),
and the game placement (`placement.rs`: the reserve beside more than 2 GiB of experts, else
2 GiB, else 0 GiB, else pause), measured once at that context like the normal one. Other
programs' memory is what the card holds after the LLM stopped, less the STT worker's model while
it is loaded (1300 MiB), read afresh at every placement. A normal placement made with no game
programs' memory is what the card holds after the LLM stopped, less the voice worker's models
while they are loaded (the STT model 1300 MiB, the TTS 1005 MiB), read afresh at every placement. A normal placement made with no game
running keeps its reading; one made while a game runs (the way back from a failed game stage)
keeps the earlier one. The game stage counts other programs as that kept reading and leaves
a reserve of at least what has grown since (the game and any program opened after that
@@ -569,7 +645,7 @@ between the normal placement and the game placement as games come and go (not wi
as the services started, there is no such reading: game detection looks once before the LLM
starts, at the window in front and at every running program (one in a launcher's game folder or
the person's games list counts, also behind other windows; how a window looks is not known
there). Then the game stage counts what the card holds now, the game included, and leaves only
there; the `game` log tab gets "a game runs as the services start: program"). Then the game stage counts what the card holds now, the game included, and leaves only
the room for other programs. The dashboard shows the stage that runs, also on a fallback
placement without experts.
- Pause: the LLM stops (`llm` off, "게임 중 일시정지"); the brain answers nothing until the stage
@@ -577,7 +653,10 @@ between the normal placement and the game placement as games come and go (not wi
- The foreground program is a game, in this order: in `notGames`: no; in `games`: yes
(`registered`); a browser (also the bot's Chrome), video player, the desktop, Discord, a
launcher's own window or this app: no; under a launcher's game folder (the folder after
`steamapps\common`, `Epic Games` or `Riot Games`) or inside a Battle.net game's install folder:
`steamapps\common`, `Epic Games` or `Riot Games`), unless that folder is the launcher's own or a
program that is not a game (`Steamworks Shared` and `wallpaper_engine` (Wallpaper Engine, which
runs all the time) under Steam, `Launcher` and `Epic Online Services` under Epic, `Riot Client`
under Riot), or inside a Battle.net game's install folder:
yes (`launcher`, with the launcher's name). Battle.net installs anywhere, so its games come from
the machine's uninstall entries (64- and 32-bit), read when the watch starts: entries removed by
its "Blizzard Uninstaller" or published by Blizzard Entertainment, with an install folder, not
@@ -602,9 +681,9 @@ start it first, so nothing answers before the brain can, and stop it last.
environment, `runtime/strata` as the working folder, one process group, lines to the `llm` log
tab). It downloads the model on the first start, starts the engine and returns; the app then
waits (up to 30 minutes) until `/v1/models` lists the model and takes its id. `--restart`
replaces a server left with other choices; the same choices are reused. `--parallel` is 1 (faster
on a 16 GB card, M0) and 2 when a chat channel answers every message, so its answers do not wait
behind voice answers. Pictures (`--vision`) are off unless screen viewing is available ("Screen viewing"): then `gpu`
replaces a server left with other choices; the same choices are reused. `--parallel` is always 1 (faster
on a 16 GB card, M0; with 2 the answers were up to 3 times slower), also when a chat channel
answers every message: its answers take turns with the voice answers. Pictures (`--vision`) are off unless screen viewing is available ("Screen viewing"): then `gpu`
in the normal stage and `cpu` in the game stage (M0). Stopping the services
runs `strata launch --stop` (ended after 2 minutes, with everything it started) and then checks
that the port refuses connections; so does a failed or cancelled start, so nothing half-started
@@ -622,7 +701,8 @@ start it first, so nothing answers before the brain can, and stop it last.
and leaves the speech-to-text worker no room, and its qwen36 engine takes no reserve. Experts get
what is left of the card after what it holds before the start (other programs), the model without
experts, the STT worker's share (2740 MiB: its model and ten speakers at once, M0 `stt`), the
engine's margin (768 MiB) and the room for programs opened later (`game.otherRoomGb`, default
TTS's share (1005 MiB, MeloTTS after its warm-up sentence, M0 `tts_melo`; it loads after the LLM,
so a start and a switch with the same programs open place the same), the engine's margin (768 MiB) and the room for programs opened later (`game.otherRoomGb`, default
1 GB); the budget first holds the long-prompt staging, rounded down to 0.1 GiB, 0 when nothing is
left. What the model holds without experts is measured once per context, number of requests at
once and card (its NVIDIA UUID): the first-run self-check, a start with `expert-vram=0`, one
@@ -636,14 +716,18 @@ start it first, so nothing answers before the brain can, and stop it last.
counted as other programs' memory; when that stop is not confirmed, nothing is read or kept and
the model starts with `expert-vram=0`. Without a card nothing is set. The game
placement (used by the switching that comes next) takes the reserve when it fits beside more
than 2 GiB of experts, else 2 GiB and the room that leaves, else 0 GiB and its room, else pause. The `llm` log tab gets the placement and what it came from.
than 2 GiB of experts, else 2 GiB and the room that leaves, else 0 GiB and its room, else pause. The `llm` log tab gets the placement and what it came from, and once the model answers how much of the card is left free (`memory.free`, without what the driver reserves: `card after the start: N MiB free of T MiB`).
- **A server the app does not start**: with `llm.url` (an OpenAI-compatible `/v1` address, in the
settings file only) the app starts nothing; that server must list a model within 30 seconds.
- **Settings**: `llm.context` (default 32768 tokens per request, 4096-262144) and `llm.port`
(default 8080) on the model screen; `llm.url` (default none).
(default 8080) on the model screen; `llm.url` (default none). `brain.location` (기본 위치, on the
model screen, default `대한민국`, up to 100 characters, empty: the brain detects it): where the
people Jarvis answers are, for the time line, weather without a place and searches
(`src/jarvis/utils/location.spec.md`).
- **The brain's settings**: on every start the app writes `config/brain.json` (it owns the file):
`ollama_base_url` the LLM's `/v1`, `ollama_chat_model` the listed id, `db_path`
`memory/jarvis.db`, `disabled_tools` and `mcps` (above, "Features and MCP servers"); the worker gets it as `JARVIS_CONFIG_PATH`. Loading it writes nothing to the
`memory/jarvis.db`, `disabled_tools` and `mcps` (above, "Features and MCP servers"),
`location_name` (`brain.location`, null when empty); the worker gets it as `JARVIS_CONFIG_PATH`. Loading it writes nothing to the
home folder.
## Chat channel names and topics (`app/src/channel_sync.rs`)
@@ -690,13 +774,16 @@ back as Korean messages for the screen. The Tauri shell passes the screens' `inv
| `discord_channel_info` `{id}` / `channel_sync_status` | a channel's Discord name and topic / each synced chat channel's last result (above) |
| `notes_list` / `note_update` `{id, text}` / `note_delete` `{id}` | everyone's personal notes (above) / change one's text / delete one |
| `history_query` `{said, answered, thought, via, person, channel, find, since, until, session, before, from, limit}` | the 대화 screen's records and answer traces (above, "Conversations") |
| `history_stamp` | `{stamp}`: changes when what the 대화 screen shows may have changed (above, "Conversations") |
| `activity_list` `{limit}` | what Jarvis did (above, "Activity"), newest first |
| `schedules_list` / `schedule_delete` `{id}` | every person's schedules (above), soonest first, with IDs as strings / delete one |
| `collect_results` `{target, limit}` | `{targets, items}`: every target folder with results, and the collected items newest first (`target` null: every target; `limit` 1-500, default 100) |
| `kma_key_status` / `kma_key_set` `{value}` | `set` and the masked form, never the key / store it (an empty value removes it); `restartNeeded` while the services run |
| `mcp_env_status` / `mcp_env_set` `{server, name, value}` | which MCP environment values are stored (`{server: {NAME: bool}}`, never the values) / store one (an empty value removes it); only for a saved server and name |
| `display_driver_status` / `display_driver_install` / `display_driver_uninstall` | the virtual monitor's driver: `{supported, installed, leftover, other, action}` (Windows only, above) / install or remove it elevated (above), then its status; refused unless the services are stopped or failed, and `service_start` / `service_restart` are refused meanwhile |
| `install_status` / `install_start` / `install_cancel` | the install steps and their state, bytes and last line / run what is not done in the background / stop it |
| `logs_query` `{tab, level, find, after, limit}` / `logs_error_report` / `logs_export` `{from, to}` | the log hub; the zip goes to `logs/` and its path comes back |
| `logs_save_text` `{text}` / `history_export` `{from, to}` / `voice_export` `{from, to}` | the 로그 screen's saves (above, "Logs"): the lines shown / the days' conversation as text (`{path, items}`) / the days' voice recordings as a zip (`{path, files, bytes}`), each in `logs/` |
| `update_status` / `update_check` / `update_download` / `update_install` | updates (below): `{current, phase, latest, error, done, total}` / look for a newer release / download it in the background / stop what runs, start the installer and answer `{exit: true}`, on which the window quits without asking |
The invite link asks for: view channels, send messages, read history, add reactions, manage
@@ -708,7 +795,8 @@ its data folder, and again after each one is saved. The log hub writes to the da
## Screens (`app/ui/`)
Plain HTML, CSS and an ES module, no build step, dark theme after `DESIGN.md`. All text is Korean.
The side menu has 대화 right under 대시보드.
The side menu has 대화 right under 대시보드 and 앱 last; there is no separate update screen (updates
are on the 앱 screen) and no separate 한 일 screen (it is a view of the 로그 screen).
- **First-run wizard** (while `firstRun`): data folder (free space, warning under 200 GB), PC check,
the model to download, Discord bot (token check with the
@@ -733,20 +821,26 @@ The side menu has 대화 right under 대시보드.
newest 100, filtered by any target that has results, each with its time, title, link and the start of its text shown
as plain text). The 예약 screen has `schedule.chatChannel` and every person's schedules (next time
or "다시 울리지 않음", repeat, who and how they asked, kind and text as plain text, the last
outcome, delete). The 앱 screen has `app.closeButton` (완전 종료 or 트레이로 최소화) with a note
that the tray icon opens the window again and its menu quits; saving it needs no restart.
outcome, delete). The 앱 screen has the update card (below) first, then `app.closeButton` (완전 종료
or 트레이로 최소화) with a note that the tray icon opens the window again and its menu quits;
saving it needs no restart.
- **사람별 지침**: every person's notes (name and ID, the text in a box, save and delete).
- **대화**: what people said, Jarvis's answers and how each answer was made, in time order with a
date line between days; people's words on the left, Jarvis's on the right with whom it answered,
each with its time, voice or chat and channel, and a copy button. A trace is a closed card (whose
question, how many tool calls, seconds, no answer) that opens to the question, the records and
notes it was given, the tools offered, the plan, the model's thinking, each tool call (name,
arguments, success, failure or skipped and why, its result) and the answer. Filters: the three
arguments, success, failure or skipped and why, its result) and the answer. A trace sits at the
time it was asked and is there while the answer is made (`pending`, "답을 만드는 중"): it fills in
step by step, and one seen being made is open and stays open with its answer unless the user
closes it. Filters: the three
kinds as chips, "현재 세션" (off by default: only items since the bot was last started, `session`;
the count line starts with that start's date and time, and with no start yet in this app run the
list says so; a new start while it is on starts the list over from that start), where (음성,
채팅, 정기 작업), person, channel, one day and text; "실시간" (on by
default) re-reads the live window every 3 seconds: every item from its first one on (`from`, up to
default) asks for `history_stamp` twice a second and re-reads the live window at once when it
changed (so a trace shows from its question on, before a quick answer ends), and every 3 seconds
anyway: every item from its first one on (`from`, up to
2,000). Only items in that window change on screen, an edited message included; items paged back
past it stay as they were loaded (edits and deletes there show after a reload of the screen). The
screen redraws only when an item changed, keeping the cards the user opened open and the scroll
@@ -757,13 +851,19 @@ The side menu has 대화 right under 대시보드.
how far back it goes; a filter change drops every answer to a request made before it, and a live
answer older than the newest one asked is dropped; "생각 모두 펼치기/접기"; a count of each kind
shown. The paging is `app/ui/talk.js`, pure, checked by `node --test app/tests/ui/talk.test.mjs`.
- **한 일**: the activity (time, kind, target, who asked, outcome and why it failed), newest 200,
with a refresh button.
- **업데이트**: the version here, the state, "다시 확인", "받기" (with its percentage) and "설치하고
다시 시작", and the release's notes as plain text. The app checks once when the main screens open;
a newer release renames the button "업데이트 (새 버전)" and says so once.
- **Logs**: tab, level and text filters, pause, live follow (lines after the last seen), "최근 오류
복사" (to the clipboard; over 2,000 characters it suggests the zip), and the zip export by days.
- **업데이트** (a card on the 앱 screen, re-read every second while it is shown): the version here,
the state, "다시 확인", "받기" (with its percentage) and "설치하고 다시 시작", and the release's
notes as plain text. The app checks once when the main screens open; a newer release renames the
앱 button "앱 (새 버전)" and says so once.
- **Logs** (로그): four views switched at the top, 로그, 대화, 목소리 and 한 일. 한 일 is the activity (time, kind,
target, who asked, outcome and why it failed), newest 200, read when the view opens, with a
refresh button; there is no separate 한 일 screen. 로그 has tab, level and text filters, pause, live follow (lines after the last seen), "최근 오류
복사" (to the clipboard; over 2,000 characters it suggests the zip), "보이는 글 txt로 저장"
(`logs_save_text` with the lines shown), and the zip export by days. 대화 saves the 대화 screen's
content of a range of days (today by default) as text (`history_export`). 목소리 has the
`voice.record` switch (saved at once; it says the bot's next start applies it, and that other
people's voices in the channel are kept too, for 7 days) and saves a range of days' recordings as a
zip (`voice_export`; over 10 MB it says Discord's file limit is passed).
`jarvis-admin-dev [--port N] [--ui DIR] [--location FILE]` (development only) serves `app/ui` and
the commands on 127.0.0.1; every command needs the random token it prints (`#token=` in the page
@@ -834,7 +934,9 @@ both to a release with that tag. The repository's other releases (v1's `v1.x`) a
`Jarvis-Setup-<version>.exe` with its `.sha256`. The installer is about 10 MB; models and runtimes
come with the first-run wizard into the data folder, which the installer never touches.
- Program Files (`Jarvis`), one administrator prompt, 64-bit Windows 10/11 only; Korean pages:
- Program Files (`Jarvis`), or where Jarvis is already installed (`HKLM\Software\JarvisBot`
`InstallDir`, 64-bit view; a folder given with `/D` wins), one administrator prompt, 64-bit
Windows 10/11 only; Korean pages:
folder, components, progress, finish ("Jarvis 실행", started through Explorer so the app runs as
the signed-in user, not as administrator).
- Components: Jarvis (always), start menu shortcut (on), desktop shortcut (off), the virtual
@@ -847,8 +949,14 @@ come with the first-run wizard into the data folder, which the installer never t
content (`jarvis-<sha256 prefix>.ico`; the old one is deleted), because Windows keeps an icon it
has shown by its path and a changed icon under the old path stays old on the desktop and in
search. After the shortcuts, every `.lnk` to `jarvis.exe` on the desktops and in the start menus
(the installing user's and every user's, the person's own and an earlier version's included)
gets this icon file, and Explorer is told to redraw (`SHCNE_ASSOCCHANGED`).
and their folders (the installing user's and every user's, the person's own and an earlier
version's included) gets this icon file (the paths reach PowerShell through the environment, so
any folder name works; a failure is a line in the details), and Explorer is told to redraw (`SHCNE_ASSOCCHANGED`).
A shortcut to the `jarvis.exe` of another folder that holds a Jarvis install (`uninstall.exe` and
`python-src` next to it: a copy left behind when an update before 0.1.12 installed into Program
Files instead of the chosen folder) is pointed at this install's `jarvis.exe` (and its working
folder at this folder), so it starts the Jarvis that is kept up to date, and a start while it runs
shows the running window instead of a second, old Jarvis. The old copy itself is not touched.
- The WebView2 runtime is installed with Microsoft's bootstrapper (inside the installer) only when
the PC has none, before any file is copied. When it is still missing afterwards the installer
asks to retry or cancel; silently it stops with exit code 3, nothing installed and nothing
@@ -945,7 +1053,8 @@ back. Performance stages (on every system, with a fake
Strata and card): a game moves the LLM to the game placement (measured once at 16K and one request,
2 GiB on the game PC's numbers, half the 16 engine threads) and back after the wait, pause stops
it, a card with no room for a game pauses, the normal stage counts a program opened since, the game
stage does not count the game twice and counts what grew since the normal reading, a game running
stage does not count the game twice and counts what grew since the normal reading, the voice
worker's models loaded after the start leave the normal placement as it started, a game running
as the services start does not pause the LLM, the way back to normal from a failed game stage while
the game runs keeps the reading from before the game, a game program behind other windows is found
at the start (win-ci VM: a minimised program in a Steam game folder), a fallback game placement still shows as the game
@@ -955,7 +1064,8 @@ is tried three times; after that a new game tries again, and the next try waits
failed switch was over. The worker client ignores a late idle of an earlier switch and sends the
latest LLM state first when a worker connects again. On every system: game detection's rules
(a borderless game, a maximised program, a browser's full-screen video, exclusive full screen,
launcher folders and their own windows, Battle.net games by install folder and their uninstall
launcher folders, their own windows and own folders (Wallpaper Engine, the Riot client's crash
handler), Battle.net games by install folder and their uninstall
entries, non-ASCII paths, the person's lists first, staying on until the game ends,
detection off) and its settings. A removal forgets the driver only when no device is left and the package
is gone or was not in the store; a failed install forgets it only under the same rule (and never

View File

@@ -12,7 +12,7 @@ use serde_json::{json, Map, Value};
use crate::control::{Controller, ServicesHost, State};
use crate::datadir::{free_bytes, DataDir, RECOMMENDED_FREE_BYTES};
use crate::logs::{Level, LogHub};
use crate::secrets::{masked, Secrets, DISCORD_TOKEN};
use crate::secrets::{masked, Secrets, DISCORD_TOKEN, KMA_KEY};
pub const APP_VERSION: &str = env!("CARGO_PKG_VERSION");
@@ -38,6 +38,17 @@ fn arg_str<'a>(args: &'a Value, key: &str) -> Result<&'a str, String> {
arg(args, key)?.as_str().ok_or_else(|| format!("{key}은(는) 글자여야 합니다"))
}
/// A `YYYY-MM-DD` day argument.
fn day_arg(args: &Value, key: &str) -> Result<chrono::NaiveDate, String> {
chrono::NaiveDate::parse_from_str(arg_str(args, key)?, "%Y-%m-%d").map_err(|_| format!("{key}는 YYYY-MM-DD 날짜여야 합니다"))
}
/// Unix seconds of a local day's start.
fn day_start(day: chrono::NaiveDate) -> f64 {
day.and_hms_opt(0, 0, 0).and_then(|t| t.and_local_timezone(chrono::Local).earliest())
.map_or(0.0, |t| t.timestamp() as f64)
}
fn level_of(s: &str) -> Level {
match s {
"error" => Level::Error,
@@ -189,7 +200,10 @@ impl AdminApi {
}
pub async fn call(&self, command: &str, args: Value) -> Result<Value, String> {
crate::debug_log!("admin", "command {command}");
// The 대화 screen asks for the stamp twice a second: a line each time would bury the log.
if command != "history_stamp" {
crate::debug_log!("admin", "command {command}");
}
match command {
"app_status" => {
let dir = self.dir.lock().unwrap().clone();
@@ -277,6 +291,19 @@ impl AdminApi {
let t = self.secrets()?.get(DISCORD_TOKEN).map_err(|e| e.to_string())?;
Ok(json!({"set": t.is_some(), "masked": t.as_deref().map(masked)}))
}
"kma_key_status" => {
let k = self.secrets()?.get(KMA_KEY).map_err(|e| e.to_string())?;
Ok(json!({"set": k.is_some(), "masked": k.as_deref().map(masked)}))
}
"kma_key_set" => {
let value = arg_str(&args, "value")?.trim();
let secrets = self.secrets()?;
if value.is_empty() { secrets.remove(KMA_KEY) } else { secrets.set(KMA_KEY, value) }.map_err(|e| e.to_string())?;
self.hub.set_secrets(secrets.values());
crate::info_log!("admin", "kma key {}", if value.is_empty() { "removed" } else { "saved" });
Ok(json!({"set": !value.is_empty(), "masked": (!value.is_empty()).then(|| masked(value)),
"restartNeeded": self.controller()?.state() == crate::control::State::Running}))
}
"discord_guilds" => Ok(json!(crate::discord_setup::guilds(&self.token()?).await?)),
"discord_channels" => {
let guild: u64 = arg_str(&args, "guildId")?.parse().map_err(|_| "서버 ID가 올바르지 않습니다")?;
@@ -405,6 +432,13 @@ impl AdminApi {
crate::info_log!("admin", "note {id} deleted");
Ok(json!({"deleted": id.to_string()}))
}
// The 대화 screen (admin.spec.md, "Conversations"): whether what it shows changed.
"history_stamp" => {
let path = self.dir()?.path().join("memory").join("history.sqlite");
let stamp = tokio::task::spawn_blocking(move || crate::history_view::stamp(&path))
.await.map_err(|e| e.to_string())??;
Ok(json!({"stamp": stamp}))
}
// The 대화 screen (admin.spec.md, "Conversations"): records and answer traces.
"history_query" => {
let dir = self.dir()?;
@@ -489,14 +523,53 @@ impl AdminApi {
Ok(json!({"report": report, "tooLongForDiscord": too_long}))
}
"logs_export" => {
let day = |k: &str| -> Result<chrono::NaiveDate, String> {
chrono::NaiveDate::parse_from_str(arg_str(&args, k)?, "%Y-%m-%d").map_err(|_| format!("{k}는 YYYY-MM-DD 날짜여야 합니다"))
};
let (from, to) = (day("from")?, day("to")?);
let (from, to) = (day_arg(&args, "from")?, day_arg(&args, "to")?);
let out = self.dir()?.logs().join(format!("jarvis-logs-{from}-{to}.zip"));
let dropped = self.hub.export_zip(from, to, &out).map_err(|e| e.to_string())?;
Ok(json!({"path": out.to_string_lossy(), "droppedLines": dropped}))
}
// The 로그 view's lines as shown, masked again, in a text file in `logs/`.
"logs_save_text" => {
let text = arg_str(&args, "text")?;
let logs = self.dir()?.logs();
std::fs::create_dir_all(&logs).map_err(|e| e.to_string())?;
let out = logs.join(format!("jarvis-log-view-{}.txt", chrono::Local::now().format("%Y%m%d-%H%M%S")));
std::fs::write(&out, crate::logs::redact(text, &self.hub.masked_values())).map_err(|e| e.to_string())?;
Ok(json!({"path": out.to_string_lossy()}))
}
// The 로그 screen's 대화 view: the days' records and traces in full, as text in `logs/`.
"history_export" => {
let (from, to) = (day_arg(&args, "from")?, day_arg(&args, "to")?);
let dir = self.dir()?;
let names: std::collections::HashMap<String, String> = crate::settings::load_settings(&dir.settings_file())
.map(|s| s.chat_channels).unwrap_or_default().into_iter()
.map(|c| (c.id.to_string(), format!("{} (채팅)", c.name))).collect();
let bot = self.secrets()?.get(DISCORD_TOKEN).ok().flatten().and_then(|t| crate::history_view::bot_id_of_token(&t));
let path = dir.path().join("memory").join("history.sqlite");
let (since, until) = (day_start(from), day_start(to + chrono::Days::new(1)));
let (body, items) = tokio::task::spawn_blocking(move || {
crate::history_view::export_text(&path, bot.as_deref(), since, until, &names)
}).await.map_err(|e| e.to_string())??;
let text = format!("Jarvis 대화 기록 {from} ~ {to} · {items}개 · 내보낸 때 {}\n\n{body}",
chrono::Local::now().format("%Y-%m-%d %H:%M:%S"));
let out = dir.logs().join(format!("jarvis-talk-{from}-{to}.txt"));
std::fs::create_dir_all(dir.logs()).map_err(|e| e.to_string())?;
std::fs::write(&out, crate::logs::redact(&text, &self.hub.masked_values())).map_err(|e| e.to_string())?;
Ok(json!({"path": out.to_string_lossy(), "items": items}))
}
// The 로그 screen's 목소리 view: the days' recordings and their listing in a zip in `logs/`.
"voice_export" => {
let (from, to) = (day_arg(&args, "from")?, day_arg(&args, "to")?);
let dir = self.dir()?;
let out = dir.logs().join(format!("jarvis-voice-{from}-{to}.zip"));
std::fs::create_dir_all(dir.logs()).map_err(|e| e.to_string())?;
let root = dir.path().join("recordings");
let o2 = out.clone();
let files = tokio::task::spawn_blocking(move || crate::recordings::export_zip(&root, from, to, &o2))
.await.map_err(|e| e.to_string())?.map_err(|e| e.to_string())?;
let bytes = std::fs::metadata(&out).map(|m| m.len()).unwrap_or(0);
Ok(json!({"path": out.to_string_lossy(), "files": files, "bytes": bytes}))
}
_ => Err(format!("알 수 없는 명령: {command}")),
}
}
@@ -603,6 +676,7 @@ mod tests {
let data = base.join("d");
a.call("folder_choose", json!({"path": data.to_string_lossy()})).await.unwrap();
assert_eq!(a.call("history_query", json!({})).await.unwrap()["missing"], true);
let before = a.call("history_stamp", json!({})).await.unwrap();
std::fs::create_dir_all(data.join("memory")).unwrap();
let c = rusqlite::Connection::open(data.join("memory").join("history.sqlite")).unwrap();
c.execute_batch(crate::history_view::TEST_SCHEMA).unwrap();
@@ -617,6 +691,46 @@ mod tests {
assert_eq!(v["channels"][0]["name"], "잡담");
let v = a.call("history_query", json!({"said": false, "find": "반가"})).await.unwrap();
assert_eq!(v["items"][0]["type"], "thought");
// The screen polls the stamp and reads again only when it changed.
let now = a.call("history_stamp", json!({})).await.unwrap();
assert_ne!(now["stamp"], before["stamp"]);
assert_eq!(a.call("history_stamp", json!({})).await.unwrap(), now);
}
#[tokio::test]
async fn the_log_screen_saves_the_shown_lines_the_days_conversation_and_the_voice_recordings() {
let (a, base) = api("api-exports");
let data = base.join("d");
a.call("folder_choose", json!({"path": data.to_string_lossy()})).await.unwrap();
a.call("token_save", json!({"token": "abc.def.ghijkl"})).await.unwrap();
// The lines as shown, with a stored secret masked even if it was on screen.
let r = a.call("logs_save_text", json!({"text": "12:00:00 INFO [app] started abc.def.ghijkl"})).await.unwrap();
let saved = std::fs::read_to_string(r["path"].as_str().unwrap()).unwrap();
assert!(saved.contains("[app] started") && !saved.contains("abc.def.ghijkl"), "{saved}");
// The conversation of a day, in full.
std::fs::create_dir_all(data.join("memory")).unwrap();
let c = rusqlite::Connection::open(data.join("memory").join("history.sqlite")).unwrap();
c.execute_batch(crate::history_view::TEST_SCHEMA).unwrap();
let now = chrono::Local::now().timestamp() as f64;
c.execute("INSERT INTO history(ts, kind, channel_id, author_id, author_name, text, norm) VALUES (?, 'voice', 'vc', '1', 'Alice', '아이유 신곡 틀어 줘', '')",
[now]).unwrap();
c.execute("INSERT INTO traces(ts, via, channel_id, user_id, question, answer, ms, steps) VALUES (?, 'voice', 'vc', '1', '아이유 신곡 틀어 줘', '틀었습니다', 900, ?)",
rusqlite::params![now + 1.0, r#"[{"kind":"tool","name":"botBrowser","args":{"action":"open"},"ok":true,"result":"Media: paused"}]"#]).unwrap();
c.execute("INSERT INTO history(ts, kind, channel_id, author_id, author_name, text, norm) VALUES (?, 'voice', 'vc', '1', 'Alice', '어제 한 말', '')",
[now - 86400.0 * 2.0]).unwrap();
let today = chrono::Local::now().date_naive().to_string();
let r = a.call("history_export", json!({"from": today, "to": today})).await.unwrap();
assert_eq!(r["items"], 2);
let talk = std::fs::read_to_string(r["path"].as_str().unwrap()).unwrap();
for want in ["Alice(1) · 음성", "아이유 신곡 틀어 줘", "도구 호출: botBrowser (성공)", "Media: paused", "답: 틀었습니다"] {
assert!(talk.contains(want), "{want} in {talk}");
}
assert!(!talk.contains("어제 한 말"));
assert!(a.call("history_export", json!({"from": "어제", "to": today})).await.is_err());
// The voice recordings of the day (none kept yet: a zip that says so).
let r = a.call("voice_export", json!({"from": today, "to": today})).await.unwrap();
assert_eq!(r["files"], 0);
assert!(std::path::Path::new(r["path"].as_str().unwrap()).starts_with(data.join("logs")));
}
#[tokio::test]
@@ -692,6 +806,28 @@ mod tests {
assert!(a.call("collect_results", json!({"target": "../config"})).await.is_err());
}
#[tokio::test]
async fn the_kma_key_is_stored_encrypted_shown_masked_kept_by_mcp_saves_and_removed_when_empty() {
let (a, base) = api("api-kma");
let data = base.join("d");
a.call("folder_choose", json!({"path": data.to_string_lossy()})).await.unwrap();
assert_eq!(a.call("kma_key_status", json!({})).await.unwrap()["set"], json!(false));
let r = a.call("kma_key_set", json!({"value": " kma-key-5678 "})).await.unwrap();
assert_eq!((r["set"].clone(), r["masked"].clone()), (json!(true), json!("••••5678")));
assert_eq!(a.call("kma_key_status", json!({})).await.unwrap(), json!({"set": true, "masked": "••••5678"}));
if cfg!(windows) {
assert!(!String::from_utf8_lossy(&std::fs::read(data.join("config").join("secrets.dat")).unwrap()).contains("kma-key-5678"));
}
a.hub.push(Level::Error, "llm", "weather said kma-key-5678");
assert!(!a.call("logs_error_report", json!({})).await.unwrap()["report"].as_str().unwrap().contains("kma-key-5678"));
// Saving the MCP servers removes only their own values.
a.call("settings_apply", json!({"changes": {"mcpServers": []}})).await.unwrap();
assert_eq!(a.call("kma_key_status", json!({})).await.unwrap()["set"], json!(true));
assert!(!std::fs::read_to_string(data.join("config").join("settings.json")).unwrap_or_default().contains("kma-key-5678"));
a.call("kma_key_set", json!({"value": ""})).await.unwrap();
assert_eq!(a.call("kma_key_status", json!({})).await.unwrap(), json!({"set": false, "masked": null}));
}
#[tokio::test]
async fn mcp_values_are_stored_encrypted_shown_only_as_set_masked_in_logs_and_go_with_their_server() {
let (a, base) = api("api-mcp");

View File

@@ -234,6 +234,9 @@ pub fn chrome_args(profile: &Path, proxy_port: u16, no_sandbox: bool, screen: bo
"--disable-default-apps".into(),
"--disable-features=Translate,MediaRouter,OptimizationHints".into(),
"--window-size=1280,800".into(),
// Video and sound start when a page starts them: the tool clicks with page scripts, which
// Chrome does not count as a user's gesture, so a video page opened for "play X" stayed paused.
"--autoplay-policy=no-user-gesture-required".into(),
];
if no_sandbox {
a.push("--no-sandbox".into());
@@ -576,7 +579,8 @@ mod tests {
fn chrome_goes_through_the_proxy_with_its_own_profile() {
let a = chrome_args(Path::new("/d/browser/profile"), 4321, false, false);
for want in ["--headless=new", "--user-data-dir=/d/browser/profile", "--remote-debugging-port=0",
"--proxy-server=http://127.0.0.1:4321", "--proxy-bypass-list=<-loopback>", "--disable-quic"] {
"--proxy-server=http://127.0.0.1:4321", "--proxy-bypass-list=<-loopback>", "--disable-quic",
"--autoplay-policy=no-user-gesture-required"] {
assert!(a.iter().any(|x| x == want), "{want}");
}
assert!(!a.iter().any(|x| x == "--no-sandbox"));
@@ -592,7 +596,7 @@ mod tests {
assert!(a.iter().any(|x| x == "--no-startup-window"));
assert!(!a.iter().any(|x| x == "about:blank"));
for want in ["--disable-backgrounding-occluded-windows", "--disable-renderer-backgrounding",
"--proxy-server=http://127.0.0.1:4321", "--disable-quic"] {
"--proxy-server=http://127.0.0.1:4321", "--disable-quic", "--autoplay-policy=no-user-gesture-required"] {
assert!(a.iter().any(|x| x == want), "{want}");
}
}

View File

@@ -20,6 +20,15 @@ pub const ADDRESSED_TIMEOUT: Duration = Duration::from_secs(5);
/// Missed requests kept for the dashboard (M5).
pub const MISSED_KEPT: usize = 100;
/// The voice channel as an "addressed" question describes it (conversation.spec.md, "Call detection").
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Room {
/// People in the channel, the speaker included (not Jarvis).
pub people: usize,
/// Of them, the people who may call Jarvis (all, or only the designated ones).
pub callers: usize,
}
/// What the conversation needs from the voice side.
pub trait Voice {
/// Have the brain answer and speak it; the speech request id, or None when it cannot. `heard`:
@@ -33,7 +42,10 @@ pub trait Voice {
fn earcon(&self);
/// A call is still being said (partial transcript): the brain may get ready to answer it.
fn prepare(&self, user: u64, text: &str);
fn ask_addressed(&self, id: &str, user: u64, text: &str) -> bool;
/// Ask the brain whether `text` was said to Jarvis, telling it who is in the channel.
fn ask_addressed(&self, id: &str, user: u64, text: &str, room: Room) -> bool;
/// The people in Jarvis's voice channel that it hears (not Jarvis, not ignored bots).
fn voice_people(&self) -> Vec<u64>;
fn ask_rank(&self, id: &str, candidates: &[(String, u64, String)]) -> bool;
/// Have the brain answer a chat message (not spoken); its text comes as `CoreEvent::ReplyText`.
fn chat_answer(&self, id: &str, user: u64, channel: u64, text: &str) -> bool;
@@ -60,8 +72,11 @@ impl Voice for DiscordCore {
fn prepare(&self, user: u64, text: &str) {
DiscordCore::prepare(self, user, text);
}
fn ask_addressed(&self, id: &str, user: u64, text: &str) -> bool {
DiscordCore::ask_addressed(self, id, user, text)
fn ask_addressed(&self, id: &str, user: u64, text: &str, room: Room) -> bool {
DiscordCore::ask_addressed(self, id, user, text, room.people, room.callers)
}
fn voice_people(&self) -> Vec<u64> {
self.voice_members().into_iter().map(|(u, _)| u).collect()
}
fn ask_rank(&self, id: &str, candidates: &[(String, u64, String)]) -> bool {
DiscordCore::ask_rank(self, id, candidates)
@@ -155,6 +170,17 @@ pub struct Conversation<V: Voice> {
}
impl<V: Voice> Conversation<V> {
/// Who is in the voice channel with the speaker (who is always counted, also while the
/// channel list lags), and how many of them may call Jarvis.
fn room(&self, speaker: u64) -> Room {
let mut people = self.voice.voice_people();
if !people.contains(&speaker) {
people.push(speaker);
}
let callers = people.iter().filter(|u| !self.calls.designated_only || self.calls.designated.contains(u)).count();
Room { people: people.len(), callers }
}
pub fn new(voice: V, calls: CallSettings, policy: PolicySettings) -> Self {
Self { voice, calls, policy: Policy::new(policy), next_id: 0, answers: HashMap::new(),
by_speech: HashMap::new(), asks: HashMap::new(), missed: MissedLog::default(), chat: ChatCalls::default(),
@@ -295,8 +321,16 @@ impl<V: Voice> Conversation<V> {
self.apply(a, now);
}
Call::Ask => {
let room = self.room(speaker);
if room.people <= 1 {
// Alone with Jarvis: whatever the speaker says is said to it.
debug_log!("conversation", "addressed req={} alone with jarvis", req.id);
let a = self.policy.request(req, now);
self.apply(a, now);
return;
}
let id = format!("q{}", req.id);
if self.voice.ask_addressed(&id, speaker, &text) {
if self.voice.ask_addressed(&id, speaker, &text, room) {
self.asks.insert(id, (req, now));
} else {
let a = self.policy.utterance_closed(speaker, now);
@@ -483,6 +517,8 @@ mod tests {
Earcon,
Prepare(u64, String),
Addressed(String, u64),
/// The room an "addressed" question described: people in the channel, people who may call.
Room(usize, usize),
Rank(String, Vec<String>),
ChatAnswer(String, u64, String),
/// The channel a chat answer was asked for.
@@ -496,6 +532,8 @@ mod tests {
did: Rc<RefCell<Vec<Did>>>,
n: Rc<RefCell<u32>>,
chat_in: Rc<RefCell<Vec<Did>>>,
/// Who is in the voice channel.
people: Rc<RefCell<Vec<u64>>>,
}
impl Voice for Fake {
fn answer(&self, user: u64, text: &str, _heard: (Instant, Instant)) -> Option<String> {
@@ -515,10 +553,14 @@ mod tests {
fn prepare(&self, user: u64, text: &str) {
self.did.borrow_mut().push(Did::Prepare(user, text.into()));
}
fn ask_addressed(&self, id: &str, user: u64, _text: &str) -> bool {
fn ask_addressed(&self, id: &str, user: u64, _text: &str, room: Room) -> bool {
self.did.borrow_mut().push(Did::Addressed(id.into(), user));
self.chat_in.borrow_mut().push(Did::Room(room.people, room.callers));
true
}
fn voice_people(&self) -> Vec<u64> {
self.people.borrow().clone()
}
fn ask_rank(&self, id: &str, c: &[(String, u64, String)]) -> bool {
self.did.borrow_mut().push(Did::Rank(id.into(), c.iter().map(|x| x.0.clone()).collect()));
true
@@ -548,6 +590,7 @@ mod tests {
let f = Fake::default();
let calls = CallSettings { mode, earcon: false, ..CallSettings::default() };
let policy = PolicySettings { kind, collect: Duration::ZERO, ..PolicySettings::default() };
*f.people.borrow_mut() = vec![A, B];
(Conversation::new(f.clone(), calls, policy), f, Instant::now())
}
fn conv_earcon(kind: PolicyKind) -> (Conversation<Fake>, Fake, Instant) {
@@ -760,6 +803,36 @@ mod tests {
assert_eq!(c.missed().filter(|m| m.reason == MissReason::Stopped).count(), 2);
}
#[test]
fn addressed_mode_answers_everything_a_person_says_alone_with_jarvis() {
let (mut c, f, t) = conv(PolicyKind::InOrder, CallMode::Addressed);
*f.people.borrow_mut() = vec![A];
c.on_event(&fin(A, "넌 무슨 말에 응답하는 거야", 100), ms(t, 0));
assert_eq!(f.take(), vec![Did::Answer(A, "넌 무슨 말에 응답하는 거야".into())], "no question to the brain");
// Not yet in the channel list (it can lag): the speaker is there all the same.
c.on_event(&end("s1", false), ms(t, 2000));
*f.people.borrow_mut() = vec![];
c.on_event(&fin(A, "오늘 부산에 비 온대", 100), ms(t, 2100));
assert_eq!(f.take(), vec![Did::Answer(A, "오늘 부산에 비 온대".into())]);
}
#[test]
fn with_others_in_the_channel_the_brain_is_told_how_many_are_there_and_may_call() {
let (mut c, f, t) = conv(PolicyKind::InOrder, CallMode::Addressed);
c.on_event(&fin(A, "불 좀 꺼 줄래", 100), ms(t, 0));
assert!(matches!(f.take()[..], [Did::Addressed(_, A)]));
assert_eq!(f.chat_in.take(), vec![Did::Room(2, 2)]);
// Only designated people may call: the others count as people, not as callers.
let f = Fake::default();
*f.people.borrow_mut() = vec![A, B, 33];
let calls = CallSettings { mode: CallMode::Addressed, earcon: false, designated: [A].into(), designated_only: true,
..CallSettings::default() };
let mut c = Conversation::new(f.clone(), calls, PolicySettings { kind: PolicyKind::InOrder, ..PolicySettings::default() });
c.on_event(&fin(A, "불 좀 꺼 줄래", 100), ms(t, 0));
assert!(matches!(f.take()[..], [Did::Addressed(_, A)]));
assert_eq!(f.chat_in.take(), vec![Did::Room(3, 1)]);
}
#[test]
fn addressed_mode_asks_the_brain_and_answers_only_a_yes() {
let (mut c, f, t) = conv(PolicyKind::InOrder, CallMode::Addressed);

View File

@@ -51,8 +51,14 @@ Decision for one utterance (`judge(speaker, text)`), in order:
and has at most 3 more letters ("멈춰", "멈춰 줘", "그만해"): **stop**.
4. A wake word is heard: **request**, `direct` (called by name).
5. `designatedNoWakeWord` and the speaker is designated: **request**, not direct.
6. `call.mode` `everything`: **request**, not direct. `addressed`: **ask** (the brain judges whether it
was said to Jarvis; a yes becomes a request, not direct). `wakeWord`: ignore.
6. `call.mode` `everything`: **request**, not direct. `addressed`: when the speaker is the only person in
Jarvis's voice channel (the people it hears, not Jarvis or ignored bots; the speaker always counts,
also while the channel list lags), **request**, not direct: alone with Jarvis, whatever they say is
said to it. Otherwise **ask**: the brain judges whether it was said to Jarvis, told how many people
are in the channel and how many of them may call Jarvis (all, or with `designatedOnly` the
designated ones present), since with one other person most of what the speaker says is for Jarvis
and with more most talk is between people (`voice_worker.spec.md`, `addressed`); a yes becomes a
request, not direct. `wakeWord`: ignore.
The text sent to the brain is the utterance as heard; the brain treats the wake word as address.
@@ -83,11 +89,12 @@ Settings (section `response`):
| Key | Default | Meaning |
|---|---|---|
| `response.policy` | `"first"` | `first` (처음 대화만), `inOrder` (들어온 순서대로), `latest` (마지막 대화만), `important` (제일 중요한 대화만). |
| `response.policy` | `"latest"` | `first` (처음 대화만), `inOrder` (들어온 순서대로), `latest` (마지막 대화만), `important` (제일 중요한 대화만). |
| `response.queueMax` | `5` | `inOrder`: most waiting requests; past it the oldest is dropped. 1-20. |
| `response.collectMs` | `700` | `important`: how long candidates are collected before choosing. 0-3000. |
| `response.settleMs` | `800` | Longest wait for an earlier-ended utterance that is still being transcribed. 0-3000. |
| `response.priorities` | `{}` | `important`: user ID → priority (higher first, default 0). |
| `response.interrupt` | `true` | `latest` and `important`: a new request may stop the running answer (below). `false`: the running answer is always spoken to its end and the new request waits for it (`latest`: the newest waiting request is answered next and older waiting ones are missed, `replaced`, also a request whose final arrives after a newer one's; `important`: the best candidate collected meanwhile is answered next, no ranking against the running answer). A voice stop still stops it. `first` and `inOrder` never interrupt. |
Terms:
@@ -110,7 +117,7 @@ Rules shared by every policy:
The stopped answer and the dropped requests are missed (`stopped`).
- An answer that is already done is never started again.
Per policy:
Per policy (with `response.interrupt` on; off, no answer is stopped for a new request):
| Policy | No answer running | A new request while an answer runs |
|---|---|---|
@@ -160,10 +167,9 @@ never takes the voice turn and never goes through the response policy.
(`reply_delivered`).
- The worker writes voice and chat answers in separate lanes (`voice_worker.spec.md`), so a voice
answer never waits behind another person's chat answer; one person's answers are written one at a
time, in the order they asked. For the two to run at the same time the LLM must take two
requests at once (`docs/v2/conversation.md`, "LLM 동시 요청 수"): until the app starts Strata itself
(M5), Strata is started with `--parallel 2` when some channel answers everything; with
`--parallel 1` the two lanes' LLM calls take turns at Strata.
time, in the order they asked. The LLM takes one request at a time (`docs/v2/conversation.md`,
"LLM 동시 요청 수"; `--parallel 1`, also when a channel answers everything), so the two lanes' LLM
calls take turns at Strata.
## The conversation loop
@@ -204,8 +210,9 @@ each to the chat model, the heard text fenced as data. Eval: `evals/test_voice_j
`cargo test` covers call detection (exact and near wake words in Hangul and Latin, a different word
two letters away, wake word anywhere, stop with and without a wake word, designated combinations,
modes) and every policy rule above with fake times (settling, ordering by end time, busy, queue limit,
interruption, ranking tie, ranking timeout, stop dropping the queue), and the conversation loop with a
fake voice side (answering and queueing, a busy drop recorded without text, voice stop, addressed yes,
interruption and `response.interrupt` off, ranking tie, ranking timeout, stop dropping the queue), and the conversation loop with a
fake voice side (answering and queueing, a busy drop recorded without text, voice stop, addressed alone
with Jarvis answered without asking, the room counts sent with the question, addressed yes,
no and timeout, the earcon before each answer, once for a wake word alone, none while busy or when
off, partials that read as a call sent to get ready (joined to a waiting wake word, none while
answering), interruption cancelling the right speech, a ranking deciding, end times from

View File

@@ -621,8 +621,8 @@ impl DiscordCore {
}
/// Ask the brain whether `text` was said to Jarvis; the answer comes as `CoreEvent::Addressed`.
pub fn ask_addressed(&self, id: &str, user: u64, text: &str) -> bool {
self.inner.worker.addressed(id, user, text)
pub fn ask_addressed(&self, id: &str, user: u64, text: &str, people: usize, callers: usize) -> bool {
self.inner.worker.addressed(id, user, text, people, callers)
}
/// Tell the brain a call from `user` is being said; nothing comes back.

View File

@@ -135,7 +135,14 @@ and the late "left" update cancelled the new join (seen on real Discord).
- **Hearing**: songbird only decrypts (transport and DAVE). Its own Opus decoding is not used: in
songbird 0.6.1 it decodes the payload without trimming what DAVE decryption left at its end, which
garbled, repeated or dropped about a third of the frames (measured with a test tone that changes
pitch every 20 ms). The core takes each decrypted RTP packet with songbird's exact payload bounds:
pitch every 20 ms). Discord names a speaker's SSRC (its Speaking op, over the websocket) only the
first time they speak, after their first packets arrive over UDP; songbird dropped the DAVE packets
it could not yet decrypt, which cut the first ~100 ms off the first thing each person said after a
join (live 2026-11-18: "자비스" heard as "아비"). The app therefore uses songbird 0.6.1 with a patch
(`app/vendor/songbird`, `JARVIS-PATCH.md`): packets of an SSRC not named yet are held, at most one
second and 50 packets for each of at most 32 SSRCs, and processed in arrival order once it is named;
the Speaking event goes out first, so the core knows the speaker before their held packets come.
The core takes each decrypted RTP packet with songbird's exact payload bounds:
- Packets go through a per-speaker reorder window in sequence-number order. A missing packet is
waited for until two later packets have arrived (about 40 ms), then rebuilt from the next packet's
Opus in-band FEC (the last missing frame) or concealed (any earlier ones). Late and duplicate
@@ -167,7 +174,9 @@ and the late "left" update cancelled the new join (seen on real Discord).
sends becomes one clip (a 48 kHz stereo WAV played through songbird without resampling); clips play
in sentence order as soon as they arrive, and requests play first come, first served. A sentence that
failed is skipped and the rest still play. `SpeakEnd` reports how many sentences played and whether
the request was cancelled or failed.
the request was cancelled or failed; the worker is told the same (`tts_played`, with `last_ms`, how
long the clip playing when it was stopped had played, timed from when it was handed to the player)
so its voice recordings keep only what was heard (`voice_worker.spec.md`, "Recordings").
- **Answering** (M4): `answer(user, text)` queues a request like `say` whose speech the worker makes
from the brain's answer (`reply` with `speak`, under the request's id) and returns its id. No answer
from the brain, a refused `reply` (`brain_unavailable`) or a lost worker ends it as failed.

View File

@@ -28,12 +28,14 @@ const NOT_GAMES: &[&str] = &[
];
/// Launcher game folders, as the path parts before the game's own folder: a program under one
/// counts as a game whatever its window looks like. Battle.net installs anywhere, so its games come
/// from their install folders (`InstalledGame`).
const LAUNCHERS: &[(&[&str], &str)] = &[
(&["steamapps", "common"], "Steam"),
(&["epic games"], "Epic"),
(&["riot games"], "Riot"),
/// counts as a game whatever its window looks like, unless that folder is one of the launcher's
/// own (its client, shared runtimes, or a non-game app such as Wallpaper Engine, which runs all
/// the time). Battle.net installs anywhere, so its games come from their install folders
/// (`InstalledGame`).
const LAUNCHERS: &[(&[&str], &str, &[&str])] = &[
(&["steamapps", "common"], "Steam", &["steamworks shared", "wallpaper_engine"]),
(&["epic games"], "Epic", &["launcher", "epic online services"]),
(&["riot games"], "Riot", &["riot client"]),
];
/// A game a launcher installed somewhere of its own choosing.
@@ -71,10 +73,11 @@ fn launcher_game(path: &str, installed: &[InstalledGame]) -> Option<(&'static st
return Some((g.launcher, g.name.clone()));
}
}
for (marker, launcher) in LAUNCHERS {
for (marker, launcher, own) in LAUNCHERS {
// The game's folder follows the marker, and the program is inside it.
if let Some(i) = (0..p.len().saturating_sub(marker.len() + 1)).find(|&i| same(&p[i..i + marker.len()], marker)) {
return Some((launcher, p[i + marker.len()].to_owned()));
let folder = p[i + marker.len()];
return (!own.iter().any(|o| folder.eq_ignore_ascii_case(o))).then(|| (*launcher, folder.to_owned()));
}
}
None
@@ -150,14 +153,14 @@ pub fn classify(fg: &Foreground, d3d_fullscreen: bool, s: &GameSettings, install
(fg.covers_monitor && fg.no_frame && fg.gpu_3d >= MIN_GPU_3D).then_some(Reason::Borderless)
}
/// Whether one of these programs (full paths) is a game by what it is: the person's list or a
/// The first of these programs (full paths) that is a game by what it is: the person's list or a
/// launcher's game folder. Used once as the services start, for games behind other windows (how a
/// window looks is not known there, so those rules do not apply).
pub fn game_running_among(paths: &[String], s: &GameSettings, installed: &[InstalledGame]) -> bool {
paths.iter().any(|p| {
let fg = Foreground { pid: 0, path: p.clone(), covers_monitor: false, no_frame: false, gpu_3d: 0.0 };
pub fn game_running_among(paths: &[String], s: &GameSettings, installed: &[InstalledGame]) -> Option<String> {
paths.iter().find(|p| {
let fg = Foreground { pid: 0, path: (*p).clone(), covers_monitor: false, no_frame: false, gpu_3d: 0.0 };
classify(&fg, false, s, installed).is_some()
})
}).cloned()
}
/// A game's name for the dashboard: its name from the launcher (the folder under a launcher's game
@@ -225,7 +228,11 @@ impl GameWatch {
if cfg!(windows) && settings.detect {
let snap = probe::Probe::new().snapshot();
state.lock().unwrap().update(&snap, &settings, probe::alive);
background = game_running_among(&probe::process_paths(), &settings, &snap.installed);
if let Some(path) = game_running_among(&probe::process_paths(), &settings, &snap.installed) {
// Which program, so a wrong one can be put on the "not games" list.
crate::info_log!("game", "a game runs as the services start: {}", file_name(&path));
background = true;
}
}
let thread = (cfg!(windows) && settings.detect).then(|| {
let (state, stop) = (state.clone(), stop.clone());
@@ -622,15 +629,35 @@ mod tests {
let s = settings();
let wow = InstalledGame { dir: r"D:\Games\World of Warcraft".into(), name: "World of Warcraft".into(), launcher: "Battle.net" };
// Behind other windows: only what the program is (where it lives, or the person's list).
assert!(game_running_among(&[r"C:\Windows\explorer.exe".into(), r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()], &s, &[]));
assert!(game_running_among(&[r"D:\Games\World of Warcraft\_retail_\Wow.exe".into()], &s, &[wow]));
assert!(!game_running_among(&[r"C:\Windows\explorer.exe".into(), r"C:\Riot Games\Riot Client\RiotClientServices.exe".into(),
r"C:\Program Files\Google\Chrome\Application\chrome.exe".into()], &s, &[]), "launchers and browsers are not games");
assert_eq!(game_running_among(&[r"C:\Windows\explorer.exe".into(), r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()], &s, &[]),
Some(r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()), "the program found, for the log");
assert!(game_running_among(&[r"D:\Games\World of Warcraft\_retail_\Wow.exe".into()], &s, &[wow]).is_some());
assert!(game_running_among(&[r"C:\Windows\explorer.exe".into(), r"C:\Riot Games\Riot Client\RiotClientServices.exe".into(),
r"C:\Program Files\Google\Chrome\Application\chrome.exe".into()], &s, &[]).is_none(), "launchers and browsers are not games");
let mut mine = settings();
mine.games = vec!["mygame".into()];
assert!(game_running_among(&[r"D:\x\MyGame.exe".into()], &mine, &[]));
assert!(game_running_among(&[r"D:\x\MyGame.exe".into()], &mine, &[]).is_some());
mine.not_games = vec!["VALORANT.exe".into()];
assert!(!game_running_among(&[r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()], &mine, &[]));
assert!(game_running_among(&[r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()], &mine, &[]).is_none());
}
#[test]
fn programs_in_a_launchers_own_folders_are_not_games() {
let s = settings();
// Always running on the game PC (2026-11-17): Wallpaper Engine is a Steam app, and the Riot
// client's crash handler sits in the client's folder.
for p in [r"C:\Program Files (x86)\Steam\steamapps\common\wallpaper_engine\wallpaper64.exe",
r"C:\Program Files (x86)\Steam\steamapps\common\Steamworks Shared\_CommonRedist\vcredist\x.exe",
r"C:\Riot Games\Riot Client\RiotClientCrashHandler.exe",
r"C:\Program Files\Epic Games\Launcher\Portal\Binaries\Win64\EpicGamesLauncher.exe",
r"C:\Program Files (x86)\Epic Games\Epic Online Services\service\EpicOnlineServices.exe"] {
assert_eq!(classify(&fg(p, false, false, 0.0), false, &s, &[]), None, "{p}");
}
assert_eq!(game_running_among(&[r"C:\Riot Games\Riot Client\RiotClientCrashHandler.exe".into(),
r"C:\Program Files (x86)\Steam\steamapps\common\wallpaper_engine\wallpaper64.exe".into()], &s, &[]), None);
// The games beside them still are.
assert_eq!(classify(&fg(r"C:\Program Files\Epic Games\Fortnite\FortniteClient.exe", false, false, 0.0), false, &s, &[]),
Some(Reason::Launcher("Epic")));
}
#[test]

View File

@@ -215,11 +215,13 @@ pub fn query(path: &Path, bot: Option<&str>, q: &Query) -> Result<Value, String>
let (id, ts): (i64, f64) = (r.get(0)?, r.get(1)?);
let user: String = r.get(4)?;
let steps: String = r.get(8)?;
let ms: i64 = r.get(7)?;
Ok(((ts, 1, id), json!({
"key": format!("t{id}"), "type": "thought", "at": ts, "via": r.get::<_, String>(2)?,
"channel": r.get::<_, Option<String>>(3)?,
"who": {"id": user.clone(), "name": name_of(&user)},
"question": r.get::<_, String>(5)?, "answer": r.get::<_, String>(6)?, "ms": r.get::<_, i64>(7)?,
"question": r.get::<_, String>(5)?, "answer": r.get::<_, String>(6)?, "ms": ms.max(0),
"pending": ms < 0,
"steps": serde_json::from_str::<Value>(&steps).unwrap_or(json!([])),
})))
}).map_err(err)?;
@@ -252,6 +254,147 @@ pub fn query(path: &Path, bot: Option<&str>, q: &Query) -> Result<Value, String>
"people": people, "channels": channels}))
}
/// Every item between `since` (inclusive) and `until` (exclusive) as plain text, in time order:
/// what people said, Jarvis's answers, and each trace in full (the records and notes it was given,
/// the tools offered, the plan, the model's thinking, every tool call with its arguments and result,
/// the answer), for the 로그 screen's 대화 export. `names`: chat channel id to the name given it.
/// Returns the text and how many items it holds.
pub fn export_text(path: &Path, bot: Option<&str>, since: f64, until: f64, names: &HashMap<String, String>)
-> Result<(String, usize), String> {
let mut pages: Vec<Vec<Value>> = vec![];
let mut before = None;
loop {
let q = Query { since: Some(since), until: Some(until), before, limit: MAX_LIMIT, ..Query::default() };
let v = query(path, bot, &q)?;
let items = v["items"].as_array().cloned().unwrap_or_default();
before = items.first().and_then(Cursor::of);
let more = v["more"].as_bool().unwrap_or(false);
pages.push(items);
if !more || before.is_none() {
break;
}
}
let items: Vec<Value> = pages.into_iter().rev().flatten().collect();
let mut out = String::new();
for i in &items {
export_item(&mut out, i, names);
}
Ok((out, items.len()))
}
fn export_item(out: &mut String, i: &Value, names: &HashMap<String, String>) {
use std::fmt::Write;
let s = |v: &Value| v.as_str().unwrap_or("").to_string();
let at = chrono::DateTime::from_timestamp_millis((i["at"].as_f64().unwrap_or(0.0) * 1000.0) as i64)
.map(|t| t.with_timezone(&chrono::Local).format("%Y-%m-%d %H:%M:%S").to_string()).unwrap_or_default();
let via = match i["via"].as_str() { Some("voice") => "음성", Some("chat") => "채팅", Some("task") => "정기 작업", _ => "" };
let channel = match i["channel"].as_str() {
Some(id) if !id.is_empty() => match names.get(id) {
Some(n) => format!(" · {n}"),
None => format!(" · {via} 채널 {id}"),
},
_ => String::new(),
};
let person = |w: &Value| {
let (id, name) = (s(&w["id"]), s(&w["name"]));
if name.is_empty() { id } else if id.is_empty() { name } else { format!("{name}({id})") }
};
let indent = |text: &str, pad: &str| text.lines().map(|l| format!("{pad}{l}\n")).collect::<String>();
match i["type"].as_str() {
Some("thought") => {
let tools = i["steps"].as_array().map_or(0, |st| st.iter()
.filter(|x| x["kind"] == "tool" && x.get("skipped").is_none_or(Value::is_null)).count());
let took = if i["pending"].as_bool() == Some(true) { "답을 만드는 중".to_string() }
else { format!("{:.1}초", i["ms"].as_f64().unwrap_or(0.0) / 1000.0) };
let _ = writeln!(out, "[{at}] 자비스 생각 · {}의 말에 답하며 · 도구 {tools}번 · {took} · {via}{channel}", person(&i["who"]));
let _ = writeln!(out, " 받은 말: {}", s(&i["question"]));
for st in i["steps"].as_array().into_iter().flatten() {
export_step(out, st, &indent);
}
let answer = s(&i["answer"]);
let _ = writeln!(out, " 답: {}", if i["pending"].as_bool() == Some(true) { "(만드는 중)".into() }
else if answer.is_empty() { "(답하지 못함)".into() } else { answer });
}
Some(kind) => {
let who = if kind == "answered" {
match i.get("to").filter(|t| !t.is_null()) { Some(t) => format!("자비스 → {}", person(t)), None => "자비스".into() }
} else { person(&i["who"]) };
let _ = writeln!(out, "[{at}] {who} · {via}{channel}");
out.push_str(&indent(&s(&i["text"]), " "));
}
None => {}
}
out.push('\n');
}
fn export_step(out: &mut String, st: &Value, indent: &dyn Fn(&str, &str) -> String) {
use std::fmt::Write;
let list = |k: &str| st[k].as_array().map(|a| a.iter().filter_map(Value::as_str).map(String::from).collect::<Vec<_>>())
.unwrap_or_default();
match st["kind"].as_str() {
Some("records") => {
let (recent, matches) = (list("recent"), list("matches"));
let notes = st["notes"].as_u64().unwrap_or(0);
let _ = writeln!(out, " 참고한 기록: 최근 대화 {}줄 · 관련 기록 {}줄 · 사람별 지침 {notes}개", recent.len(), matches.len());
for (title, xs) in [("최근 대화", recent), ("관련 기록", matches)] {
if !xs.is_empty() {
let _ = writeln!(out, " {title}:");
out.push_str(&indent(&xs.join("\n"), " "));
}
}
}
Some("tools") => { let _ = writeln!(out, " 쓸 수 있게 고른 도구: {}", list("names").join(", ")); }
Some("plan") => {
let _ = writeln!(out, " 계획:");
for (n, x) in list("steps").iter().enumerate() {
let _ = writeln!(out, " {}. {x}", n + 1);
}
}
Some("think") => {
let _ = writeln!(out, " 모델 생각:");
out.push_str(&indent(st["text"].as_str().unwrap_or(""), " "));
}
Some("tool") => {
let state = match st["skipped"].as_str() {
Some("duplicate") => "건너뜀: 같은 호출을 되풀이해 부르지 않음".to_string(),
Some("not_allowed") => "건너뜀: 허용되지 않은 도구라 부르지 않음".to_string(),
Some(other) => format!("건너뜀: {other}"),
None => (if st["ok"].as_bool() == Some(true) { "성공" } else { "실패" }).to_string(),
};
let _ = writeln!(out, " 도구 호출: {} ({state})", st["name"].as_str().unwrap_or(""));
if st["args"].as_object().is_some_and(|a| !a.is_empty()) {
let _ = writeln!(out, " 인자:");
out.push_str(&indent(&serde_json::to_string_pretty(&st["args"]).unwrap_or_default(), " "));
}
if let Some(r) = st["result"].as_str().filter(|r| !r.is_empty()) {
let _ = writeln!(out, " 결과:");
out.push_str(&indent(r, " "));
}
}
other => { let _ = writeln!(out, " {}: {st}", other.unwrap_or("?")); }
}
}
/// A cheap mark of what the 대화 screen shows that changes when something is said or a question
/// arrives (the newest record and trace), and as an answer is made (its steps) and ends, so the
/// screen can poll it often and read the items again only when it changed.
pub fn stamp(path: &Path) -> Result<String, String> {
if !path.is_file() {
return Ok("missing".into());
}
let c = open(path)?;
let err = |e: rusqlite::Error| format!("대화 기록을 읽지 못했습니다: {e}");
let said: Option<i64> = c.query_row("SELECT MAX(id) FROM history", [], |r| r.get(0)).map_err(err)?;
let thought: (Option<i64>, i64, Option<i64>) = if has_traces(&c) {
c.query_row("SELECT MAX(id), (SELECT COUNT(*) FROM traces WHERE ms < 0), \
(SELECT SUM(LENGTH(steps)) FROM traces WHERE ms < 0) FROM traces", [], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))
.map_err(err)?
} else {
(None, 0, None)
};
Ok(format!("{said:?}/{thought:?}"))
}
/// The history store's tables as `src/jarvis/memory/history.py` makes them, for tests.
#[cfg(test)]
pub(crate) const TEST_SCHEMA: &str = "
@@ -384,6 +527,71 @@ mod tests {
assert!(Cursor::of(&json!({"at": 1.0, "key": "x1"})).is_none());
}
#[test]
fn a_trace_still_being_made_reads_as_pending_with_its_steps_so_far() {
let (path, c) = db("pending");
rec(&c, 10.0, "voice", "vc", "1", "Alice", None, "서울 날씨 어때");
c.execute("INSERT INTO traces(ts, via, channel_id, user_id, question, answer, ms, steps) VALUES (?,?,?,?,?,?,?,?)",
rusqlite::params![10.5, "voice", "vc", "1", "서울 날씨 어때", "", -1, r#"[{"kind":"tools","names":["getWeather"]}]"#])
.unwrap();
let v = query(&path, None, &Query::default()).unwrap();
let t = &v["items"][1];
assert_eq!((t["type"].as_str(), t["pending"].as_bool()), (Some("thought"), Some(true)));
assert_eq!(t["steps"][0]["names"], json!(["getWeather"]));
let done = query(&sample("pending-done"), None, &Query::default()).unwrap();
assert_eq!(done["items"][1]["pending"], false);
}
#[test]
fn the_stamp_changes_when_a_question_is_answered_step_by_step_and_only_then() {
let (path, c) = db("stamp");
let now = || stamp(&path).unwrap();
let empty = now();
assert_eq!(now(), empty, "nothing changed");
rec(&c, 10.0, "voice", "vc", "1", "Alice", None, "서울 날씨 어때");
let said = now();
assert_ne!(said, empty);
c.execute("INSERT INTO traces(ts, via, channel_id, user_id, question, answer, ms, steps) \
VALUES (10.5, 'voice', 'vc', '1', '서울 날씨 어때', '', -1, '[]')", []).unwrap();
let asked = now();
assert_ne!(asked, said, "a trace still being made");
c.execute("UPDATE traces SET steps = '[{\"kind\":\"tools\",\"names\":[\"getWeather\"]}]'", []).unwrap();
let step = now();
assert_ne!(step, asked, "a step added");
c.execute("UPDATE traces SET answer = '맑습니다', ms = 900", []).unwrap();
assert_ne!(now(), step, "the answer");
let dir = std::env::temp_dir().join(format!("jarvis-history-view-nostamp-{}", std::process::id()));
assert!(stamp(&dir.join("history.sqlite")).is_ok());
}
#[test]
fn the_export_holds_every_item_of_the_days_in_order_with_each_trace_in_full() {
let path = sample("export");
let c = Connection::open(&path).unwrap();
trace(&c, 25.0, "voice", Some("vc"), "1", "아이유 신곡 틀어 줘", "",
r#"[{"kind":"records","recent":["[14:44] Alice: 안녕"],"matches":[],"notes":2},
{"kind":"plan","steps":["botBrowser open"]},
{"kind":"tool","name":"botBrowser","args":{"action":"open","url":"https://youtube.com"},"ok":true,"result":"Title: YouTube\nMedia: paused"},
{"kind":"tool","name":"webSearch","args":{"q":"x"},"ok":false,"skipped":"duplicate"}]"#);
drop(c);
let names = HashMap::from([("ch".to_string(), "잡담".to_string())]);
let (text, n) = export_text(&path, Some("900"), 0.0, 100.0, &names).unwrap();
assert_eq!(n, 8);
let order: Vec<usize> = ["서울 날씨 어때", " 답: 맑습니다", "자비스 → Alice(1) · 음성", "Bob(2) · 채팅 · 잡담",
"아이유 신곡 틀어 줘", "약속 알림"].iter().map(|x| text.find(x).unwrap_or_else(|| panic!("{x} in\n{text}"))).collect();
assert!(order.windows(2).all(|w| w[0] < w[1]), "time order:\n{text}");
for want in ["쓸 수 있게 고른 도구: getWeather", "모델 생각:\n 날씨 도구를 쓰자", " [14:44] Alice: 안녕",
"사람별 지침 2개", " 1. botBrowser open", "도구 호출: botBrowser (성공)", "\"url\": \"https://youtube.com\"",
" 결과:\n Title: YouTube\n Media: paused", "도구 호출: webSearch (건너뜀: 같은 호출을 되풀이해 부르지 않음)",
"도구 1번", " 답: (답하지 못함)"] {
assert!(text.contains(want), "{want:?} in\n{text}");
}
// Only the days asked for.
let (text, n) = export_text(&path, Some("900"), 20.0, 23.0, &names).unwrap();
assert_eq!(n, 3);
assert!(!text.contains("서울 날씨"));
}
#[test]
fn the_bot_id_comes_from_its_token() {
assert_eq!(bot_id_of_token("MTIzNDU2Nzg5MDEyMzQ1Njc4.GxYzAb.abcdef").as_deref(), Some("123456789012345678"));

View File

@@ -30,6 +30,7 @@ pub mod permissions;
pub mod placement;
pub mod policy;
pub mod receive;
pub mod recordings;
pub mod schedule;
pub mod screen;
pub mod secrets;

View File

@@ -13,6 +13,15 @@ pub const MARGIN_MIB: i64 = 768;
pub const STT_MIB: i64 = STT_MODEL_MIB + 1440;
/// The STT worker's model alone (M0 `stt`).
pub const STT_MODEL_MIB: i64 = 1300;
/// The TTS (MeloTTS, Korean) on the card after its warm-up sentence, measured on the game PC
/// (M0 `tts_melo`). It loads after the LLM starts, so it is kept beside like the STT's share.
pub const TTS_MIB: i64 = 1005;
/// What the voice worker's models hold on the card now: the STT model and the TTS once loaded.
/// Taken off what the card holds, so they are not counted again as other programs.
pub fn worker_models_mib(stt_loaded: bool, tts_loaded: bool) -> i64 {
(if stt_loaded { STT_MODEL_MIB } else { 0 }) + if tts_loaded { TTS_MIB } else { 0 }
}
/// The smallest expert budget in a game: it holds the long-prompt staging (about 1.1 GiB) first,
/// so 2 GiB is the least that loads experts, and it was 1.5-2.1 times faster than none in a real
/// game (user decision, 2026-11-10).
@@ -36,10 +45,10 @@ pub struct Probe {
}
/// The expert budget (GiB, down to 0.1) that leaves `room_mib` free besides what the card already
/// holds (`used_mib`), the LLM without experts, the STT's share and the engine's margin. The budget
/// holds (`used_mib`), the LLM without experts, the STT's and TTS's shares and the engine's margin. The budget
/// first holds the staging, which the start without experts already has (M0 `expert_budget_gib`).
pub fn expert_budget_gib(total_mib: i64, used_mib: i64, p: &Probe, room_mib: i64) -> f64 {
let spare = total_mib - used_mib - p.model_mib - STT_MIB - MARGIN_MIB - room_mib;
let spare = total_mib - used_mib - p.model_mib - STT_MIB - TTS_MIB - MARGIN_MIB - room_mib;
if spare <= 0 {
return 0.0;
}
@@ -71,7 +80,7 @@ pub fn game(total_mib: i64, used_mib: i64, p: &Probe, reserve_mib: i64) -> GameP
return GamePlacement { expert_gib: expert, game_room_mib: reserve_mib, paused: false };
}
// Beside the model without experts (which already holds the staging): what 2 GiB adds.
let free = total_mib - used_mib - p.model_mib - STT_MIB - MARGIN_MIB;
let free = total_mib - used_mib - p.model_mib - STT_MIB - TTS_MIB - MARGIN_MIB;
let extra = ((MIN_GAME_EXPERT_GIB - p.staging_gib) * 1024.0).max(0.0) as i64;
if free - extra > 0 {
return GamePlacement { expert_gib: MIN_GAME_EXPERT_GIB, game_room_mib: free - extra, paused: false };
@@ -93,7 +102,7 @@ mod tests {
use super::*;
/// The game PC in M0 `gamemode_tune` (RTX 5070 Ti 16 GB): 2760 MiB held before the LLM there
/// included the STT model (1300 MiB), which the app starts after the LLM.
/// included the STT model (1300 MiB), which the app starts after the LLM. M0 ran no TTS.
const TOTAL: i64 = 16303;
const OTHERS: i64 = 2760 - 1300;
fn probe32() -> Probe {
@@ -103,7 +112,8 @@ mod tests {
#[test]
fn the_normal_placement_is_the_one_m0_measured_with_room_for_the_stt() {
// M0 got 1.5 GiB (the STT model counted as already held); the app counts it in the STT share.
assert_eq!(normal(TOTAL, OTHERS, &probe32(), 1024), 1.5);
// M0 ran no TTS: the app keeps its share too, so the same 1.5 GiB needs that much less held.
assert_eq!(normal(TOTAL, OTHERS - TTS_MIB, &probe32(), 1024), 1.5);
// More room for other programs, fewer experts; no room at all, none.
assert!(normal(TOTAL, OTHERS, &probe32(), 2048) < 1.5);
assert_eq!(normal(TOTAL, OTHERS + 8000, &probe32(), 1024), 0.0);
@@ -116,7 +126,7 @@ mod tests {
// 8 GB does not fit on 16 GB: the smallest placement, and the game gets what is left.
let g = game(TOTAL, OTHERS, &probe16, 8192);
assert_eq!((g.expert_gib, g.paused), (MIN_GAME_EXPERT_GIB, false));
let expected = TOTAL - OTHERS - probe16.model_mib - STT_MIB - MARGIN_MIB - ((2.0 - 1.1) * 1024.0) as i64;
let expected = TOTAL - OTHERS - probe16.model_mib - STT_MIB - TTS_MIB - MARGIN_MIB - ((2.0 - 1.1) * 1024.0) as i64;
assert_eq!(g.game_room_mib, expected);
assert!(g.game_room_mib > 0 && g.game_room_mib < 8192);
// A bigger card keeps the whole reserve, with more experts than the minimum.
@@ -124,9 +134,9 @@ mod tests {
assert_eq!(big.game_room_mib, 8192);
assert!(big.expert_gib > MIN_GAME_EXPERT_GIB && !big.paused);
// No room beside 2 GiB but some beside none: the 0 GiB placement, before pausing.
let tight = game(TOTAL, 2960, &Probe { model_mib: 9124, ..probe16.clone() }, 8192);
let tight = game(TOTAL, 1955, &Probe { model_mib: 9124, ..probe16.clone() }, 8192);
assert_eq!((tight.expert_gib, tight.paused), (0.0, false));
assert_eq!(tight.game_room_mib, TOTAL - 2960 - 9124 - STT_MIB - MARGIN_MIB);
assert_eq!(tight.game_room_mib, TOTAL - 1955 - 9124 - STT_MIB - TTS_MIB - MARGIN_MIB);
// Other programs already hold the card: no room for a game, the LLM pauses in one.
let full = game(TOTAL, OTHERS + 4000, &probe16, 8192);
assert!(full.paused);

View File

@@ -13,6 +13,10 @@ pub trait TtsBackend {
/// Send text for request `id`; false when the worker is not connected (the request then fails).
fn tts(&self, id: &str, text: &str, more: bool) -> bool;
fn tts_cancel(&self, id: &str);
/// Request `id` ended after `played` of its sentences started playing, the last of them for
/// `last_ms` before it was stopped (None: played to its end). The worker's recording keeps that
/// much (voice_worker.spec.md, "Recordings").
fn tts_played(&self, id: &str, played: u32, cancelled: bool, last_ms: Option<u32>);
}
/// Plays one PCM clip at a time; its owner calls `Playback::clip_ended()` when the clip finishes.
@@ -53,11 +57,13 @@ pub struct Playback<B: TtsBackend, S: ClipSink> {
playing: bool,
counter: u64,
events: Vec<PlaybackEvent>,
/// When the clip now playing started.
clip_started: Option<std::time::Instant>,
}
impl<B: TtsBackend, S: ClipSink> Playback<B, S> {
pub fn new(backend: B, sink: S) -> Self {
Self { backend, sink, queue: VecDeque::new(), playing: false, counter: 0, events: vec![] }
Self { backend, sink, queue: VecDeque::new(), playing: false, counter: 0, events: vec![], clip_started: None }
}
/// True while a clip is playing or a request is waiting for its audio.
@@ -120,13 +126,14 @@ impl<B: TtsBackend, S: ClipSink> Playback<B, S> {
if !r.done {
self.backend.tts_cancel(&r.id);
}
let mut last_ms = None;
if pos == 0 && self.playing {
self.playing = false;
last_ms = self.clip_elapsed_ms();
self.sink.stop();
}
crate::debug_log!("play", "cancel id={} played={}", r.id, r.played);
self.events.push(PlaybackEvent::End(PlaybackEnd { id: r.id, played: r.played, cancelled: true,
failed: false }));
self.ended(PlaybackEnd { id: r.id, played: r.played, cancelled: true, failed: false }, last_ms);
self.pump();
}
@@ -164,6 +171,7 @@ impl<B: TtsBackend, S: ClipSink> Playback<B, S> {
pub fn stop(&mut self) -> usize {
let stopped: Vec<Request> = self.queue.drain(..).collect();
let was_playing = self.playing;
let head_ms = if was_playing { self.clip_elapsed_ms() } else { None };
self.playing = false;
for r in stopped.iter().filter(|r| !r.done) {
self.backend.tts_cancel(&r.id);
@@ -171,14 +179,25 @@ impl<B: TtsBackend, S: ClipSink> Playback<B, S> {
if was_playing {
self.sink.stop();
}
for r in &stopped {
self.events.push(PlaybackEvent::End(PlaybackEnd { id: r.id.clone(), played: r.played, cancelled: true,
failed: false }));
for (i, r) in stopped.iter().enumerate() {
let last_ms = if i == 0 { head_ms } else { None };
self.ended(PlaybackEnd { id: r.id.clone(), played: r.played, cancelled: true, failed: false }, last_ms);
}
crate::debug_log!("play", "stop requests={} was_playing={was_playing}", stopped.len());
stopped.len()
}
/// A request ended: tell the worker how much of it was played, then the owner.
fn ended(&mut self, end: PlaybackEnd, last_ms: Option<u32>) {
self.backend.tts_played(&end.id, end.played, end.cancelled, last_ms);
self.events.push(PlaybackEvent::End(end));
}
/// How long the clip now playing has played.
fn clip_elapsed_ms(&self) -> Option<u32> {
self.clip_started.map(|t| t.elapsed().as_millis().min(u32::MAX as u128) as u32)
}
fn pump(&mut self) {
while !self.playing {
let Some(head) = self.queue.front_mut() else { return };
@@ -196,6 +215,7 @@ impl<B: TtsBackend, S: ClipSink> Playback<B, S> {
self.events.push(PlaybackEvent::Start(head.id.clone()));
}
self.playing = true;
self.clip_started = Some(std::time::Instant::now());
self.sink.play(pcm);
return;
}
@@ -204,8 +224,7 @@ impl<B: TtsBackend, S: ClipSink> Playback<B, S> {
}
let r = self.queue.pop_front().expect("head exists");
crate::debug_log!("play", "said id={} sentences={} failed={}", r.id, r.played, r.failed);
self.events.push(PlaybackEvent::End(PlaybackEnd { id: r.id, played: r.played, cancelled: false,
failed: r.failed }));
self.ended(PlaybackEnd { id: r.id, played: r.played, cancelled: false, failed: r.failed }, None);
}
}
}
@@ -221,6 +240,7 @@ mod tests {
cancelled: Vec<String>,
played: Vec<String>,
stops: usize,
reported: Vec<(String, u32, bool, Option<u32>)>,
}
#[derive(Clone, Default)]
struct Fake(Rc<RefCell<Log>>);
@@ -232,6 +252,9 @@ mod tests {
fn tts_cancel(&self, id: &str) {
self.0.borrow_mut().cancelled.push(id.into());
}
fn tts_played(&self, id: &str, played: u32, cancelled: bool, last_ms: Option<u32>) {
self.0.borrow_mut().reported.push((id.into(), played, cancelled, last_ms));
}
}
impl ClipSink for Fake {
fn play(&mut self, pcm: Vec<u8>) {
@@ -274,6 +297,35 @@ mod tests {
assert!(!p.busy());
}
#[test]
fn the_worker_is_told_how_much_of_each_request_was_played() {
let (mut p, f) = setup();
let a = p.say("하나. 둘.", false);
p.audio(&a, 0, b("a0"));
p.audio(&a, 1, b("a1"));
p.done(&done(&a, false));
p.clip_ended();
p.clip_ended();
let c = p.say("셋. 넷.", false);
p.audio(&c, 0, b("c0"));
p.audio(&c, 1, b("c1"));
p.done(&done(&c, false));
std::thread::sleep(std::time::Duration::from_millis(60));
p.cancel(&c); // stopped about 60 ms into its first sentence
let e = p.say("여섯.", false);
p.audio(&e, 0, b("e0"));
let d = p.say("다섯.", false);
p.stop(); // e stopped at once, d before any sound
let r = f.0.borrow().reported.clone();
assert_eq!(r[0], (a, 2, false, None), "played to its end");
let (id, n, cancelled, ms) = r[1].clone();
assert_eq!((id, n, cancelled), (c, 1, true));
assert!(ms.is_some_and(|ms| (50..1000).contains(&ms)), "how long the stopped sentence played: {ms:?}");
assert_eq!((r[2].0.clone(), r[2].1, r[2].2), (e, 1, true));
assert!(r[2].3.is_some_and(|ms| ms < 50));
assert_eq!(r[3], (d, 0, true, None));
}
#[test]
fn requests_play_in_the_order_they_were_made() {
let (mut p, f) = setup();

View File

@@ -28,17 +28,21 @@ pub struct PolicySettings {
pub priorities: HashMap<u64, i64>,
/// How long the brain's importance ranking is waited for.
pub rank_timeout: Duration,
/// `latest` and `important`: a new request may stop the running answer. Off: the answer is
/// spoken to its end and the new request waits for it.
pub interrupt: bool,
}
impl Default for PolicySettings {
fn default() -> Self {
Self {
kind: PolicyKind::First,
kind: PolicyKind::Latest,
queue_max: 5,
collect: Duration::from_millis(700),
settle: Duration::from_millis(800),
priorities: HashMap::new(),
rank_timeout: Duration::from_millis(1500),
interrupt: true,
}
}
}
@@ -162,8 +166,13 @@ impl Policy {
PolicyKind::First => out.push(Action::Missed(req, MissReason::Busy)),
PolicyKind::InOrder => self.enqueue(req, &mut out),
PolicyKind::Latest => {
if req.ended_at > cur.ended_at {
self.interrupt(&mut out);
// Newer than the running answer and than every waiting request (only possible
// without interrupting: otherwise the newest is already the running one).
let newest = self.waiting.iter().map(|r| r.ended_at).max().map_or(cur.ended_at, |w| w.max(cur.ended_at));
if req.ended_at > newest {
if self.s.interrupt {
self.interrupt(&mut out);
}
for r in self.waiting.drain(..) {
out.push(Action::Missed(r, MissReason::Replaced));
}
@@ -175,7 +184,9 @@ impl Policy {
PolicyKind::Important => {
let (new, running) = (self.score(&req), self.score(&cur));
self.enqueue(req, &mut out);
if new > running {
if !self.s.interrupt {
// Kept as a candidate; the best one is answered when the running answer ends.
} else if new > running {
self.interrupt(&mut out);
self.collect_until = None;
self.ranking = None;
@@ -547,6 +558,40 @@ mod tests {
assert_eq!(t.p.current(), Some(b));
}
#[test]
fn without_interrupting_latest_finishes_the_answer_then_answers_the_newest_waiting() {
let mut t = T::with(PolicySettings { kind: PolicyKind::Latest, interrupt: false, ..PolicySettings::default() });
let (a, _) = t.call(A, true, 0, 100);
let (b, out) = t.call(B, true, 500, 700);
assert!(stopped(&out).is_empty(), "the answer keeps playing");
assert!(answered(&out).is_empty());
let (c, out) = t.call(C, true, 800, 900);
assert_eq!(missed(&out), vec![(b, MissReason::Replaced)], "a newer waiting request replaces it");
assert_eq!(answered(&t.p.answer_done(a, t.at(2000))), vec![c]);
}
#[test]
fn without_interrupting_latest_keeps_the_newest_waiting_request_whatever_order_finals_arrive_in() {
let mut t = T::with(PolicySettings { kind: PolicyKind::Latest, interrupt: false, ..PolicySettings::default() });
let (a, _) = t.call(A, true, 0, 100);
let (b, _) = t.call(B, true, 700, 800);
// C ended before B but its final came later: B stays, C is replaced.
let (c, out) = t.call(C, true, 500, 900);
assert_eq!(missed(&out), vec![(c, MissReason::Replaced)]);
assert_eq!(answered(&t.p.answer_done(a, t.at(2000))), vec![b]);
}
#[test]
fn without_interrupting_important_waits_for_the_answer_even_for_a_better_request() {
let s = PolicySettings { kind: PolicyKind::Important, collect: Duration::ZERO, interrupt: false,
priorities: HashMap::from([(B, 5)]), ..PolicySettings::default() };
let mut t = T::with(s);
let (a, _) = t.call(A, true, 0, 100);
let (b, out) = t.call(B, true, 500, 700);
assert!(stopped(&out).is_empty() && ranks(&out).is_empty(), "no stop and no ranking against the answer");
assert_eq!(answered(&t.p.answer_done(a, t.at(2000))), vec![b]);
}
#[test]
fn latest_answers_the_newest_of_several_waiting() {
let mut t = T::new(PolicyKind::Latest);

115
app/src/recordings.rs Normal file
View File

@@ -0,0 +1,115 @@
//! The voice recordings the worker keeps when `voice.record` is on (`voice_worker.spec.md`,
//! "Recordings"), packed for the 로그 screen's 목소리 view (`admin.spec.md`, "Logs").
//!
//! `recordings/<YYYY-MM-DD>/` holds one WAV per utterance or spoken answer and `index.jsonl`, one
//! line per file: `{file, at, ms, kind, who, name, channel, text, cancelled}`.
use std::io::Write;
use std::path::Path;
use chrono::NaiveDate;
use serde_json::Value;
/// The recordings of days `from..=to` in a zip at `out`: every day's WAV files under `<day>/`, its
/// `index.jsonl`, and `index.txt` listing every file in time order with who said it and its text.
/// Returns how many sound files it holds.
pub fn export_zip(root: &Path, from: NaiveDate, to: NaiveDate, out: &Path) -> std::io::Result<usize> {
let mut days: Vec<(NaiveDate, std::path::PathBuf)> = std::fs::read_dir(root).into_iter().flatten().flatten()
.filter_map(|e| {
let day = NaiveDate::parse_from_str(&e.file_name().to_string_lossy(), "%Y-%m-%d").ok()?;
(day >= from && day <= to && e.path().is_dir()).then(|| (day, e.path()))
})
.collect();
days.sort();
let mut zip = zip::ZipWriter::new(std::fs::File::create(out)?);
let deflate = zip::write::SimpleFileOptions::default().compression_method(zip::CompressionMethod::Deflated);
let stored = zip::write::SimpleFileOptions::default().compression_method(zip::CompressionMethod::Stored);
let (mut listing, mut files) = (String::new(), 0usize);
for (day, dir) in &days {
let index = std::fs::read_to_string(dir.join("index.jsonl")).unwrap_or_default();
let mut entries: Vec<Value> = index.lines().filter_map(|l| serde_json::from_str(l).ok()).collect();
entries.sort_by(|a, b| a["at"].as_f64().unwrap_or(0.0).total_cmp(&b["at"].as_f64().unwrap_or(0.0)));
for e in &entries {
listing += &line_of(day, e);
}
let mut wavs: Vec<_> = std::fs::read_dir(dir)?.flatten().map(|e| e.path())
.filter(|p| p.extension().is_some_and(|x| x == "wav")).collect();
wavs.sort();
for p in wavs {
let name = format!("{day}/{}", p.file_name().unwrap_or_default().to_string_lossy());
zip.start_file(name, stored).map_err(std::io::Error::other)?;
zip.write_all(&std::fs::read(&p)?)?;
files += 1;
}
if !index.is_empty() {
zip.start_file(format!("{day}/index.jsonl"), deflate).map_err(std::io::Error::other)?;
zip.write_all(index.as_bytes())?;
}
}
zip.start_file("index.txt", deflate).map_err(std::io::Error::other)?;
zip.write_all(if listing.is_empty() { "저장된 목소리가 없습니다.\n".as_bytes() } else { listing.as_bytes() })?;
zip.finish().map_err(std::io::Error::other)?;
Ok(files)
}
fn line_of(day: &NaiveDate, e: &Value) -> String {
let s = |k: &str| e[k].as_str().unwrap_or("").to_string();
let at = chrono::DateTime::from_timestamp_millis((e["at"].as_f64().unwrap_or(0.0) * 1000.0) as i64)
.map(|t| t.with_timezone(&chrono::Local).format("%H:%M:%S%.3f").to_string()).unwrap_or_default();
let who = if s("kind") == "spoke" {
format!("자비스{}", if e["cancelled"].as_bool() == Some(true) { " (중간에 멈춤)" } else { "" })
} else if s("name").is_empty() || s("name") == s("who") {
s("who")
} else {
format!("{}({})", s("name"), s("who"))
};
let text = s("text");
format!("{day} {at} · {:.1}초 · {day}/{} · {who}: {}\n", e["ms"].as_f64().unwrap_or(0.0) / 1000.0, s("file"),
if text.trim().is_empty() { "(받아 적은 글 없음)".into() } else { text })
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Read;
#[test]
fn the_zip_holds_the_days_asked_for_with_a_listing_of_who_said_what() {
let root = std::env::temp_dir().join(format!("jarvis-recordings-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&root);
for (day, lines) in [
("2026-11-15", vec![r#"{"file":"235959-000-said-1.wav","at":1.0,"ms":800,"kind":"said","who":"1","name":"Alice","text":"어제 말"}"#]),
("2026-11-16", vec![
r#"{"file":"144843-500-jarvis.wav","at":20.0,"ms":2500,"kind":"spoke","who":"jarvis","text":"알려드리겠습니다.","cancelled":true}"#,
r#"{"file":"144842-100-said-1.wav","at":10.0,"ms":1200,"kind":"said","who":"1","name":"Alice","text":"자비스 내일 알려 줘"}"#,
r#"{"file":"144850-000-said-1.wav","at":30.0,"ms":300,"kind":"said","who":"1","name":"Alice","text":""}"#]),
] {
let dir = root.join(day);
std::fs::create_dir_all(&dir).unwrap();
std::fs::write(dir.join("index.jsonl"), lines.join("\n") + "\n").unwrap();
for l in lines {
let v: Value = serde_json::from_str(l).unwrap();
std::fs::write(dir.join(v["file"].as_str().unwrap()), b"RIFF").unwrap();
}
}
let out = root.join("voice.zip");
let d = |s: &str| NaiveDate::parse_from_str(s, "%Y-%m-%d").unwrap();
assert_eq!(export_zip(&root, d("2026-11-16"), d("2026-11-16"), &out).unwrap(), 3);
let mut z = zip::ZipArchive::new(std::fs::File::open(&out).unwrap()).unwrap();
let mut names: Vec<String> = (0..z.len()).map(|i| z.by_index(i).unwrap().name().unwrap().to_string()).collect();
names.sort();
assert_eq!(names, ["2026-11-16/144842-100-said-1.wav", "2026-11-16/144843-500-jarvis.wav",
"2026-11-16/144850-000-said-1.wav", "2026-11-16/index.jsonl", "index.txt"]);
let mut listing = String::new();
z.by_name("index.txt").unwrap().read_to_string(&mut listing).unwrap();
let lines: Vec<&str> = listing.lines().collect();
assert_eq!(lines.len(), 3);
assert!(lines[0].ends_with("1.2초 · 2026-11-16/144842-100-said-1.wav · Alice(1): 자비스 내일 알려 줘"), "{listing}");
assert!(lines[1].ends_with("2026-11-16/144843-500-jarvis.wav · 자비스 (중간에 멈춤): 알려드리겠습니다."), "{listing}");
assert!(lines[2].ends_with("Alice(1): (받아 적은 글 없음)"), "{listing}");
// No recordings at all: a zip that says so.
let none = root.join("none");
assert_eq!(export_zip(&none, d("2026-11-16"), d("2026-11-16"), &out).unwrap(), 0);
std::fs::remove_dir_all(&root).ok();
}
}

View File

@@ -9,6 +9,8 @@ use crate::datadir::write_atomic;
/// The Discord bot token's name in the store.
pub const DISCORD_TOKEN: &str = "discordToken";
/// The KMA API hub key (기상청 API허브) for the weather inside Korea.
pub const KMA_KEY: &str = "kmaApiKey";
#[derive(Debug)]
pub struct SecretsError(pub String);

View File

@@ -143,6 +143,7 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
// The MCP servers' environment values stay encrypted on disk: they reach the brain only in the
// worker's environment (admin.spec.md, "Features and MCP servers").
let mcp_env = crate::features::mcp_env(&secrets, &settings.mcp_servers)?;
let kma_key = secrets.get(crate::secrets::KMA_KEY)?;
let token = match stored {
Some(t) => t,
None => std::env::var("JARVIS_DISCORD_TOKEN").map_err(|_| "no bot token: set it in the app or JARVIS_DISCORD_TOKEN")?,
@@ -151,7 +152,7 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
// What these services are given stays masked in the logs until the next start, also if it is
// replaced or removed in the store meanwhile (admin.spec.md, "Logs").
if let Some(hub) = crate::logs::installed() {
hub.set_secrets_in_use(secrets_in_use(&token, &mcp_env));
hub.set_secrets_in_use(secrets_in_use(&token, &mcp_env, kma_key.as_deref()));
}
let (base_url, model) = llm.start(&settings).await?;
@@ -170,9 +171,15 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
}
wopts.env.insert("JARVIS_CONFIG_PATH".into(), brain_config.to_string_lossy().into_owned());
wopts.env.insert(crate::features::MCP_ENV_VAR.into(), mcp_env.to_string());
if let Some(k) = &kma_key {
wopts.env.insert(KMA_ENV_VAR.into(), k.clone());
}
if let BrowserRun::Running(b, _) = &browser {
wopts.env.insert("JARVIS_BROWSER_CDP".into(), b.cdp_url());
}
if let Some(dir) = recordings_dir(data_dir, &settings) {
wopts.env.insert(RECORDINGS_ENV_VAR.into(), dir.to_string_lossy().into_owned());
}
// Background collection (admin.spec.md): the enabled targets, and the keep days in any case.
match crate::collect::write_config(data_dir, &settings) {
Ok(f) => { wopts.env.insert(crate::collect::COLLECT_ENV_VAR.into(), f.to_string_lossy().into_owned()); }
@@ -233,6 +240,16 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
Ok(Services { core, events, worker, worker_process, missed, llm, browser, game, perf, schedules, news })
}
/// The voice worker writes its recordings where this names (voice_worker.spec.md, "Recordings").
pub const RECORDINGS_ENV_VAR: &str = "JARVIS_RECORDINGS_DIR";
/// The stored KMA key reaches the brain's weather tool only in the worker's environment.
pub const KMA_ENV_VAR: &str = "JARVIS_KMA_API_KEY";
/// Where the voice worker keeps its recordings, when `voice.record` is on.
pub fn recordings_dir(data_dir: &Path, settings: &crate::settings::Settings) -> Option<std::path::PathBuf> {
settings.voice.record.then(|| data_dir.join("recordings"))
}
/// The brain as the performance stages see it: told the LLM's state, and asked to finish what it is
/// answering (the worker client; a fake in tests).
#[async_trait::async_trait]
@@ -242,8 +259,8 @@ pub trait LlmBrain: Send + Sync {
/// `ready` (with whether a game runs) or `paused`.
fn llm_state(&self, state: &str, game: bool) -> bool;
async fn wait_llm_idle(&self, id: u64, t: std::time::Duration) -> bool;
/// The STT worker's model is loaded (on the graphics card).
fn stt_on_card(&self) -> bool;
/// What the worker's loaded models (STT, TTS) hold on the graphics card, in MiB.
fn worker_models_mib(&self) -> i64;
}
#[async_trait::async_trait]
@@ -257,8 +274,8 @@ impl LlmBrain for WorkerClient {
async fn wait_llm_idle(&self, id: u64, t: std::time::Duration) -> bool {
WorkerClient::wait_llm_idle(self, id, t).await
}
fn stt_on_card(&self) -> bool {
self.hello().is_some_and(|h| h.stt_ready)
fn worker_models_mib(&self) -> i64 {
self.hello().map_or(0, |h| crate::placement::worker_models_mib(h.stt_ready, h.tts_ready))
}
}
@@ -309,7 +326,7 @@ async fn perf_loop(settings: crate::settings::Settings, llm: Arc<Llm>, brain: Ar
if !brain.wait_llm_idle(id, std::time::Duration::from_secs(120)).await {
crate::warn_log!("perf", "Warning: the brain did not finish its answers in 2 minutes; switching anyway");
}
let ended = match llm.switch(&settings, want, brain.stt_on_card(), game()).await {
let ended = match llm.switch(&settings, want, brain.worker_models_mib(), game()).await {
Ok(ended) => {
retry.succeeded();
actual = want;
@@ -317,7 +334,7 @@ async fn perf_loop(settings: crate::settings::Settings, llm: Arc<Llm>, brain: Ar
}
Err(e) => {
crate::error_log!("perf", "the LLM did not start in the {want:?} stage: {e}; going back to {actual:?}");
let back = match llm.switch(&settings, actual, brain.stt_on_card(), game()).await {
let back = match llm.switch(&settings, actual, brain.worker_models_mib(), game()).await {
Ok(ended) => ended,
Err(e) => {
crate::error_log!("perf", "the LLM did not come back in the {actual:?} stage either: {e}");
@@ -338,8 +355,9 @@ async fn perf_loop(settings: crate::settings::Settings, llm: Arc<Llm>, brain: Ar
/// The secret values the services get: the bot token and the MCP servers' environment values.
pub fn secrets_in_use(token: &str, mcp_env: &Value) -> Vec<String> {
pub fn secrets_in_use(token: &str, mcp_env: &Value, kma_key: Option<&str>) -> Vec<String> {
let mut v = vec![token.to_owned()];
v.extend(kma_key.map(str::to_owned));
for vars in mcp_env.as_object().into_iter().flat_map(|m| m.values()) {
v.extend(vars.as_object().into_iter().flat_map(|m| m.values()).filter_map(Value::as_str).map(str::to_owned));
}
@@ -385,8 +403,8 @@ mod perf_tests {
self.0.lock().unwrap().push("waited".into());
true
}
fn stt_on_card(&self) -> bool {
false
fn worker_models_mib(&self) -> i64 {
0
}
}
@@ -486,7 +504,7 @@ mod perf_tests {
#[cfg(unix)]
#[tokio::test]
async fn going_back_to_normal_while_the_game_runs_keeps_the_reading_from_before_the_game() {
// 24 GB card; other programs 1460 MiB at the start. A game (2500 MiB) starts; the first
// 24 GB card; other programs `OTHERS` at the start. A game (2500 MiB) starts; the first
// game start fails and the normal stage comes back while the game still runs; the second
// try works. It must count other programs from the reading before the game, not one that
// has the game in it.
@@ -497,17 +515,17 @@ mod perf_tests {
let exe = crate::strata::tests::fake_strata(&l, &format!(
"case \"$*\" in *'--context 16384'*'expert-vram=0'*) ;; *'--context 16384'*) if [ ! -e '{fo}' ]; then touch '{fo}'; exit 4; fi;; esac; rm -f '{g}'; echo 'strata-q36: (1.10 GiB staging) on 16 CPU threads' >> '{log}'; echo 'Strata is serving.'",
fo = failed_once.display(), g = l.data.join("gone").display(), log = log.display()));
let others = Arc::new(std::sync::atomic::AtomicI64::new(1460));
let others = Arc::new(std::sync::atomic::AtomicI64::new(crate::strata::tests::OTHERS));
let calls = l.data.join("calls");
let o2 = others.clone();
let llm = Arc::new(Llm::with_exe(l.clone(), exe).with_gpu(Arc::new(move || {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
Some(crate::strata::Gpu { id: "GPU-test".into(), used_mib: o2.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total_mib: 24576 })
Some(crate::strata::tests::gpu("GPU-test", o2.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, 24576))
})));
let s = crate::settings::parse_settings(&serde_json::json!({"llm": {"port": port}})).unwrap();
llm.start(&s).await.unwrap();
others.store(1460 + 2500, Ordering::SeqCst);
others.store(crate::strata::tests::OTHERS + 2500, Ordering::SeqCst);
let brain = Arc::new(FakeBrain::default());
let task = tokio::spawn(perf_loop(s, llm.clone(), brain.clone(), Arc::new(|| true), Duration::from_millis(300)));
for _ in 0..100 {
@@ -518,8 +536,8 @@ mod perf_tests {
assert!(failed_once.exists(), "the first game start failed");
assert_eq!(llm.stage_view()["stage"], serde_json::json!("gaming"));
let probe16 = crate::placement::Probe { context: 16384, parallel: 1, gpu: "GPU-test".into(), model_mib: 9874, staging_gib: 1.1 };
let right = crate::placement::game(24576, 1460, &probe16, 8192).expert_gib;
let wrong = crate::placement::game(24576, 1460 + 2500, &probe16, 8192).expert_gib;
let right = crate::placement::game(24576, crate::strata::tests::OTHERS, &probe16, 8192).expert_gib;
let wrong = crate::placement::game(24576, crate::strata::tests::OTHERS + 2500, &probe16, 8192).expert_gib;
assert_ne!(right, wrong);
assert!(launches(&l).last().unwrap().contains(&format!("--set expert-vram={right} ")), "{:?}", launches(&l));
}
@@ -540,8 +558,8 @@ mod perf_tests {
async fn wait_llm_idle(&self, _id: u64, _t: Duration) -> bool {
true
}
fn stt_on_card(&self) -> bool {
false
fn worker_models_mib(&self) -> i64 {
0
}
}
@@ -577,10 +595,19 @@ mod tests {
use super::*;
#[test]
fn the_services_secrets_are_the_token_and_every_mcp_value() {
fn the_services_secrets_are_the_token_every_mcp_value_and_the_kma_key() {
let env = serde_json::json!({"brave": {"BRAVE_API_KEY": "k1"}, "other": {"A": "k2", "B": "k3"}});
let mut v = secrets_in_use("tok", &env);
let mut v = secrets_in_use("tok", &env, Some("kma"));
v.sort();
assert_eq!(v, vec!["k1", "k2", "k3", "tok"]);
assert_eq!(v, vec!["k1", "k2", "k3", "kma", "tok"]);
assert_eq!(secrets_in_use("tok", &env, None).len(), 4);
}
#[test]
fn the_worker_gets_a_recordings_folder_in_the_data_folder_only_with_voice_record_on() {
let off = crate::settings::parse_settings(&serde_json::json!({})).unwrap();
assert_eq!(recordings_dir(Path::new("/d"), &off), None);
let on = crate::settings::parse_settings(&serde_json::json!({"voice": {"record": true}})).unwrap();
assert_eq!(recordings_dir(Path::new("/d"), &on), Some(Path::new("/d").join("recordings")));
}
}

View File

@@ -55,6 +55,8 @@ pub struct VoiceSettings {
pub max_speakers: usize,
/// Transcribe other bots too (music bots are ignored by default).
pub listen_to_bots: bool,
/// Keep each utterance and spoken answer as a sound file (voice_worker.spec.md, "Recordings").
pub record: bool,
/// Posted in the voice channel's chat on every join (privacy.md).
pub join_notice: String,
/// Wake words: the STT hint and the voice wake words of call detection (`conversation.spec.md`).
@@ -224,8 +226,14 @@ pub struct Settings {
/// The window's close button (X) hides it to the tray instead of quitting (`app.closeButton`:
/// "tray" or "quit", default "quit"; admin.spec.md, "Window and Windows build").
pub close_to_tray: bool,
/// Where the people Jarvis answers are, by default (`brain.location`, default "대한민국"): the
/// brain's location for the time line, weather without a place and searches (admin.spec.md,
/// "LLM"); None (an empty value) leaves it to the brain's own detection.
pub location: Option<String>,
}
pub const DEFAULT_LOCATION: &str = "대한민국";
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SettingsError(pub String);
@@ -251,6 +259,7 @@ impl Default for Settings {
voice: VoiceSettings {
max_speakers: 10,
listen_to_bots: false,
record: false,
join_notice: DEFAULT_JOIN_NOTICE.into(),
wake_words: vec!["자비스".into()],
stt: Map::new(),
@@ -276,6 +285,7 @@ impl Default for Settings {
type_text: Level::Ask, run: Level::Ask,
screen_exclude: ["1Password", "Bitwarden", "KeePass", "KeePassXC"].map(String::from).into() },
close_to_tray: false,
location: Some(DEFAULT_LOCATION.into()),
}
}
}
@@ -437,7 +447,7 @@ fn parse_call(raw: &Map<String, Value>, wake_words: &[String], d: &CallSettings)
fn parse_response(raw: &Map<String, Value>, d: &PolicySettings) -> Result<PolicySettings, SettingsError> {
let r = section(raw, "response")?;
let kind = match get_str(r, "policy", "response", "first")?.as_str() {
let kind = match get_str(r, "policy", "response", "latest")?.as_str() {
"first" => PolicyKind::First,
"inOrder" => PolicyKind::InOrder,
"latest" => PolicyKind::Latest,
@@ -471,6 +481,7 @@ fn parse_response(raw: &Map<String, Value>, d: &PolicySettings) -> Result<Policy
settle: get_ms(r, "settleMs", "response", d.settle, 3000)?,
priorities,
rank_timeout: d.rank_timeout,
interrupt: get_bool(r, "interrupt", "response", d.interrupt)?,
})
}
@@ -579,6 +590,7 @@ pub fn parse_settings(raw: &Value) -> Result<Settings, SettingsError> {
voice: VoiceSettings {
max_speakers,
listen_to_bots: get_bool(voice, "listenToBots", "voice", d.voice.listen_to_bots)?,
record: get_bool(voice, "record", "voice", d.voice.record)?,
join_notice,
wake_words,
stt,
@@ -627,6 +639,13 @@ pub fn parse_settings(raw: &Value) -> Result<Settings, SettingsError> {
"quit" => false,
_ => return err("app.closeButton must be tray or quit"),
},
location: {
let l = get_str(section(raw, "brain")?, "location", "brain", DEFAULT_LOCATION)?.trim().to_owned();
if l.chars().count() > 100 {
return err("brain.location must be at most 100 characters");
}
(!l.is_empty()).then_some(l)
},
})
}
@@ -895,6 +914,11 @@ mod tests {
assert_eq!(load_settings(&dir.join("nope.json")).unwrap(), Settings::default());
}
#[test]
fn by_default_only_the_latest_call_is_answered() {
assert_eq!(parse_settings(&json!({})).unwrap().response.kind, PolicyKind::Latest);
}
#[test]
fn reads_a_full_file_with_bom() {
let dir = std::env::temp_dir().join(format!("jarvis-settings-full-{}", std::process::id()));
@@ -902,7 +926,7 @@ mod tests {
let path = dir.join("settings.json");
let doc = json!({
"discord": {"guildId": ID, "voiceChannelId": ID, "commandName": "jarvis", "ephemeralReplies": false},
"voice": {"maxSpeakers": 3, "listenToBots": true, "joinNotice": "알림", "wakeWords": [" 자비스 ", "", "javis"],
"voice": {"maxSpeakers": 3, "listenToBots": true, "record": true, "joinNotice": "알림", "wakeWords": [" 자비스 ", "", "javis"],
"stt": {"end_silence_ms": 250}},
"chatChannels": [{"id": ID, "name": "일정", "description": "일정 올리는 곳", "send": false}],
"voiceWorker": {"python": "/venv/bin/python", "cwd": "/repo/src", "args": ["--melo-python", "/m/python"]},
@@ -915,6 +939,8 @@ mod tests {
command_name: "jarvis".into(), ephemeral_replies: false });
assert_eq!(s.voice.max_speakers, 3);
assert!(s.voice.listen_to_bots);
assert!(s.voice.record);
assert!(!parse_settings(&json!({})).unwrap().voice.record, "off by default");
assert_eq!(s.voice.join_notice, "알림");
assert_eq!(s.voice.wake_words, vec!["자비스", "javis"]);
assert_eq!(s.voice.stt.get("end_silence_ms"), Some(&json!(250)));
@@ -955,7 +981,7 @@ mod tests {
"call": {"similarity": 0.3, "mode": "addressed", "designated": [ID], "designatedOnly": true,
"designatedNoWakeWord": true, "stopWords": ["멈춰", " "], "chatWakeWords": ["봇"],
"followUpMs": 2500, "earcon": false},
"response": {"policy": "important", "queueMax": 3, "collectMs": 500, "settleMs": 0,
"response": {"policy": "important", "queueMax": 3, "collectMs": 500, "settleMs": 0, "interrupt": false,
"priorities": {ID: 2}}
})).unwrap();
assert_eq!(s.call.wake_words, vec!["자비스", "javis"]);
@@ -972,6 +998,8 @@ mod tests {
assert_eq!(s.response.queue_max, 3);
assert_eq!(s.response.collect, Duration::from_millis(500));
assert_eq!(s.response.settle, Duration::ZERO);
assert!(!s.response.interrupt);
assert!(parse_settings(&json!({})).unwrap().response.interrupt, "on by default");
assert_eq!(s.response.priorities.get(&IDN), Some(&2));
let s = parse_settings(&json!({"chatChannels": [{"id": ID, "answer": "everything"}]})).unwrap();
assert!(s.chat_channels[0].answer_all);
@@ -984,6 +1012,7 @@ mod tests {
(json!({"call": {"earcon": "yes"}}), "call.earcon"),
(json!({"response": {"policy": "random"}}), "response.policy"),
(json!({"response": {"queueMax": 0}}), "response.queueMax"),
(json!({"response": {"interrupt": "yes"}}), "response.interrupt"),
(json!({"response": {"collectMs": 5000}}), "response.collectMs"),
(json!({"response": {"priorities": {"x": 1}}}), "response.priorities"),
];
@@ -996,6 +1025,12 @@ mod tests {
#[test]
fn the_close_button_choice_is_read_and_checked() {
assert!(!parse_settings(&json!({})).unwrap().close_to_tray, "closing quits by default");
assert_eq!(parse_settings(&json!({})).unwrap().location.as_deref(), Some(DEFAULT_LOCATION));
assert_eq!(parse_settings(&json!({"brain": {"location": " 부산 "}})).unwrap().location.as_deref(), Some("부산"));
assert_eq!(parse_settings(&json!({"brain": {"location": " "}})).unwrap().location, None, "empty: the brain detects it");
for bad in [json!({"brain": {"location": 1}}), json!({"brain": {"location": "가".repeat(101)}}), json!({"brain": 1})] {
assert!(parse_settings(&bad).unwrap_err().0.contains("brain"), "{bad}");
}
assert!(parse_settings(&json!({"app": {"closeButton": "tray"}})).unwrap().close_to_tray);
assert!(!parse_settings(&json!({"app": {"closeButton": "quit"}})).unwrap().close_to_tray);
for bad in [json!({"app": {"closeButton": "hide"}}), json!({"app": {"closeButton": 1}}), json!({"app": 1})] {

View File

@@ -94,11 +94,13 @@ pub fn fields() -> Vec<Field> {
f("call.followUpMs", Channels, Int { min: 0, max: 5000 }, json!(1500), "호출어 뒤 이어 말하기 대기(ms)"),
f("call.earcon", Channels, Bool, json!(true), "부르면 효과음"),
f("response.policy", Channels, Choice { values: &["first", "inOrder", "latest", "important"] },
json!("first"), "동시 대화 응답 정책"),
json!("latest"), "동시 대화 응답 정책"),
f("response.queueMax", Channels, Int { min: 1, max: 20 }, json!(5), "기다리는 요청 최대 개수"),
f("response.collectMs", Channels, Int { min: 0, max: 3000 }, json!(700), "중요도 비교 대기(ms)"),
f("response.settleMs", Channels, Int { min: 0, max: 3000 }, json!(800), "먼저 끝난 말 전사 대기(ms)"),
f("response.priorities", Channels, IdNumberMap, json!({}), "사람별 우선순위"),
f("response.interrupt", Channels, Bool, json!(true),
"말하는 중에 새 요청이 오면 하던 말 멈추기(가장 최근 것, 중요한 것 먼저)"),
f("voice.joinNotice", Channels, Text { max: 2000, non_empty: true }, json!(DEFAULT_JOIN_NOTICE),
"입장 시 전사 고지 문구"),
f("chatChannels", Chat, ChatChannels, json!([]), "채팅 채널"),
@@ -114,6 +116,8 @@ pub fn fields() -> Vec<Field> {
f("memory.recent_records", Memory, Int { min: 0, max: 200 }, json!(20), "최근 대화 원문 개수"),
f("memory.matches", Memory, Int { min: 0, max: 20 }, json!(5), "과거 기록 가져올 개수"),
f("memory.budget_tokens", Memory, Number { min: 0.0, max: 100000.0 }, json!(4000), "참고 분량 상한(토큰)"),
f("voice.record", Memory, Bool, json!(false),
"목소리 녹음 저장(사람 말과 자비스 말, 분석용, 7일 보관, 다음 시작부터)"),
// The worker's STT settings and their defaults (voice_worker.spec.md, "STT").
f("voice.stt.end_silence_ms", Model, Int { min: 100, max: 2000 }, json!(300), "말 끝 판정 시간(ms)"),
f("voice.stt.partials", Model, Bool, json!(true), "중간 전사"),
@@ -124,6 +128,8 @@ pub fn fields() -> Vec<Field> {
f("voice.tts.speed", Model, Number { min: 0.5, max: 2.0 }, json!(1.5), "말하기 속도(1.0 보통, 기본 1.5, 저장하면 바로 적용)"),
// The LLM (admin.spec.md, "LLM"); `llm.url` (a server the app does not start) is file-only.
f("llm.context", Model, Int { min: 4096, max: 262144 }, json!(32768), "대화 모델이 한 번에 기억하는 양(토큰)"),
f("brain.location", Model, Text { max: 100, non_empty: false }, json!(crate::settings::DEFAULT_LOCATION),
"기본 위치(날씨, 검색, 추천의 기준, 비우면 자동 감지)"),
f("llm.port", Model, Int { min: 1024, max: 65535 }, json!(8080), "대화 모델 포트"),
// The brain's features (admin.spec.md, "Features and MCP servers").
f("features.weather", Features, Bool, json!(true), "날씨"),

View File

@@ -4,7 +4,7 @@
//! shows how far it got. With `llm.url` the app uses that server instead and starts nothing.
use std::path::Path;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::{AtomicBool, AtomicI64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
@@ -41,12 +41,6 @@ pub struct LlmStatus {
pub note: Option<String>,
}
/// Requests answered at once: one is faster on a 16 GB card (M0), two only when a chat channel
/// answers every message (so its answers do not wait behind the voice answers).
pub fn parallel(settings: &Settings) -> u32 {
if settings.chat_channels.iter().any(|c| c.answer_all) { 2 } else { 1 }
}
/// `strata launch` for the services: the M0 kit's normal placement, models in `models/llm`, this
/// computer only. No pictures yet (nothing in v2 sends one; reading them takes graphics memory).
/// `--restart` replaces a server started with other choices; the same choices are reused.
@@ -68,8 +62,9 @@ pub fn launch_args(layout: &Layout, settings: &Settings, conf: &Conf, sets: &[St
a
}
/// The context and requests at once of a stage: the settings' in the normal stage; in a game 16K
/// (or less when the settings ask for less) and one request at a time.
/// The context and requests at once of a stage: the settings' context in the normal stage, in a
/// game 16K (or less when the settings ask for less); always one request at a time (faster on a
/// 16 GB card, M0; chat answers take turns with the voice answers).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Conf {
pub context: u32,
@@ -82,7 +77,7 @@ pub fn conf(settings: &Settings, stage: Stage) -> Conf {
let vision = vision(stage, crate::screen::available(settings));
match stage {
Stage::Gaming => Conf { context: crate::placement::GAME_CONTEXT.min(settings.llm.context), parallel: 1, vision },
_ => Conf { context: settings.llm.context, parallel: parallel(settings), vision },
_ => Conf { context: settings.llm.context, parallel: 1, vision },
}
}
@@ -121,6 +116,8 @@ pub struct Gpu {
pub id: String,
pub used_mib: i64,
pub total_mib: i64,
/// What is left for programs (`memory.free`): the total less what is used and what the driver reserves.
pub free_mib: i64,
}
/// Reads the card (`nvidia-smi`); None without one.
@@ -128,7 +125,7 @@ pub type GpuReader = Arc<dyn Fn() -> Option<Gpu> + Send + Sync>;
fn nvidia_memory() -> Option<Gpu> {
let mut cmd = std::process::Command::new("nvidia-smi");
cmd.args(["--query-gpu=uuid,memory.used,memory.total", "--format=csv,noheader,nounits"]);
cmd.args(["--query-gpu=uuid,memory.used,memory.total,memory.free", "--format=csv,noheader,nounits"]);
#[cfg(windows)]
{
use std::os::windows::process::CommandExt;
@@ -137,7 +134,8 @@ fn nvidia_memory() -> Option<Gpu> {
let out = cmd.output().ok()?;
let line = String::from_utf8_lossy(&out.stdout).lines().next()?.to_owned();
let mut f = line.split(',').map(str::trim);
Some(Gpu { id: f.next()?.to_owned(), used_mib: f.next()?.parse().ok()?, total_mib: f.next()?.parse().ok()? })
Some(Gpu { id: f.next()?.to_owned(), used_mib: f.next()?.parse().ok()?, total_mib: f.next()?.parse().ok()?,
free_mib: f.next()?.parse().ok()? })
}
/// The newest engine log under Strata's folder, its end only (the placement lines of the last start).
@@ -250,6 +248,7 @@ pub fn write_brain_config(data: &Path, base_url: &str, model: &str, settings: &c
"db_path": data.join("memory").join("jarvis.db").to_string_lossy(),
"disabled_tools": crate::features::disabled_tools(settings),
"mcps": crate::features::brain_mcps(&settings.mcp_servers),
"location_name": settings.location,
"_config_version": 1,
});
let mut text = serde_json::to_string_pretty(&cfg).unwrap();
@@ -278,9 +277,9 @@ pub struct Llm {
/// A game runs (or may run) as the LLM is placed: set before `start` and at every `switch`.
/// Only a normal placement made while none runs keeps its reading.
game_running: AtomicBool,
/// The STT worker's model is on the card (true at switches while the worker runs), so it is not
/// taken for another program's memory.
stt_on_card: AtomicBool,
/// What the voice worker's models (STT, TTS) hold on the card (set at switches while the worker
/// runs), so it is not taken for another program's memory.
worker_models_mib: AtomicI64,
/// The graphics memory a game gets in the placement being started.
room: Mutex<Option<i64>>,
/// The engine's CPU threads in the normal stage; a game gets half.
@@ -306,7 +305,7 @@ impl Llm {
pub fn with_exe(layout: Layout, exe: std::path::PathBuf) -> Self {
Self { layout, exe, status: Mutex::new(LlmStatus { state: LlmState::Off, managed: false, percent: None, note: None }),
active: Mutex::new(None), cancel: Arc::new(AtomicBool::new(false)), port: Mutex::new(0),
stop_timeout: Duration::from_secs(120), gpu: None, baseline: Mutex::new(None), game_running: AtomicBool::new(false), stt_on_card: AtomicBool::new(false), room: Mutex::new(None), threads: Mutex::new(None),
stop_timeout: Duration::from_secs(120), gpu: None, baseline: Mutex::new(None), game_running: AtomicBool::new(false), worker_models_mib: AtomicI64::new(0), room: Mutex::new(None), threads: Mutex::new(None),
stage: Mutex::new((Stage::Normal, None)) }
}
@@ -366,11 +365,10 @@ impl Llm {
return Ok(Some(sets));
}
let Some(card) = self.read_gpu().await else { return Ok(Some(vec![])) };
// Other programs: what the card holds without this LLM and without the STT worker's model.
// Other programs: what the card holds without this LLM and without the voice worker's models.
// The normal stage reads it afresh (programs opened since count); the game stage takes it
// from the latest normal placement, as the card then also holds the game.
let stt_model = if self.stt_on_card.load(Ordering::SeqCst) { crate::placement::STT_MODEL_MIB } else { 0 };
let now_others = (card.used_mib - stt_model).max(0);
let now_others = (card.used_mib - self.worker_models_mib.load(Ordering::SeqCst)).max(0);
// Only a normal placement made while no game runs keeps its reading (a game's memory would be
// taken for other programs'; also on the way back from a failed game stage).
if stage != Stage::Gaming && !self.game_running.load(Ordering::SeqCst) {
@@ -469,19 +467,20 @@ impl Llm {
// A new run: other programs' memory is read afresh.
*self.baseline.lock().unwrap() = None;
*self.stage.lock().unwrap() = (Stage::Normal, None);
self.stt_on_card.store(false, Ordering::SeqCst);
self.worker_models_mib.store(0, Ordering::SeqCst);
self.start_stage(settings, Stage::Normal).await.map(|r| r.expect("the normal stage never pauses"))
}
/// Move the running LLM to `stage` (`perf.rs`): a restart with that stage's placement, or a stop
/// when it pauses (also a game stage with no room for the game). Returns the stage it ended in.
/// With `llm.url` nothing moves.
/// `stt_on_card`: the STT worker's model is loaded on the card; `game_running`: a game runs now.
pub async fn switch(&self, settings: &Settings, stage: Stage, stt_on_card: bool, game_running: bool) -> Result<Stage, String> {
/// `worker_models_mib`: what the voice worker's loaded models hold on the card
/// (`placement::worker_models_mib`); `game_running`: a game runs now.
pub async fn switch(&self, settings: &Settings, stage: Stage, worker_models_mib: i64, game_running: bool) -> Result<Stage, String> {
if settings.llm.url.is_some() {
return Ok(Stage::Normal);
}
self.stt_on_card.store(stt_on_card, Ordering::SeqCst);
self.worker_models_mib.store(worker_models_mib, Ordering::SeqCst);
self.game_running.store(game_running, Ordering::SeqCst);
if stage != Stage::Paused && self.start_stage(settings, stage).await?.is_some() {
return Ok(stage);
@@ -510,6 +509,11 @@ impl Llm {
Ok(None) => {}
Ok(Some((base, model))) => {
crate::info_log!("llm", "ready: {base} model={model}");
// What the card has left for everything else once the model is in (the voice
// worker's models, a game): a slow answer later can be matched to it.
if let Some(g) = self.read_gpu().await {
crate::info_log!("llm", "card after the start: {} MiB free of {} MiB", g.free_mib, g.total_mib);
}
self.set(LlmState::Ready, None);
}
Err(e) if e == "cancelled" => self.set(LlmState::Off, None),
@@ -653,6 +657,15 @@ pub(crate) mod tests {
use crate::datadir::temp_dir;
use crate::settings::{parse_settings, ChatChannelSettings};
/// Other programs on the fake cards: the game PC's 1460 MiB in M0 less the TTS's share, which
/// the app keeps beside and M0 did not run, so M0's placements come out.
/// A card reading as `nvidia-smi` gives it, with nothing reserved by the driver.
pub(crate) fn gpu(id: &str, used_mib: i64, total_mib: i64) -> Gpu {
Gpu { id: id.into(), used_mib, total_mib, free_mib: total_mib - used_mib }
}
pub(crate) const OTHERS: i64 = 1460 - crate::placement::TTS_MIB;
pub(crate) fn layout(tag: &str) -> Layout {
Layout { data: temp_dir(tag), src: std::path::PathBuf::from("src") }
}
@@ -686,12 +699,12 @@ pub(crate) mod tests {
}
#[test]
fn a_chat_channel_answering_everything_gets_a_second_request_slot() {
fn the_llm_takes_one_request_at_a_time_also_with_a_chat_channel_answering_everything() {
let mut s = Settings::default();
assert_eq!(parallel(&s), 1);
s.chat_channels.push(ChatChannelSettings { id: 1, name: String::new(), description: String::new(), read: true,
send: true, edit: true, react: true, delete: false, pin: false, answer_all: true, sync_discord: true });
assert_eq!(parallel(&s), 2);
let a = launch_args(&layout("llm-one"), &s, &conf(&s, Stage::Normal), &[]);
assert_eq!(a[a.iter().position(|x| x == "--parallel").unwrap() + 1], "1");
}
#[test]
@@ -716,6 +729,11 @@ pub(crate) mod tests {
assert!(Path::new(v["db_path"].as_str().unwrap()).starts_with(&l.data));
assert_eq!(v["disabled_tools"], json!(crate::features::disabled_tools(&s)));
assert_eq!(v["mcps"], crate::features::brain_mcps(&s.mcp_servers));
assert_eq!(v["location_name"], crate::settings::DEFAULT_LOCATION, "the people are in Korea by default");
let none = parse_settings(&json!({"brain": {"location": ""}})).unwrap();
let f = write_brain_config(&l.data, "http://127.0.0.1:8080/v1", "m", &none).unwrap();
let v: Value = serde_json::from_str(&std::fs::read_to_string(f).unwrap()).unwrap();
assert_eq!(v["location_name"], Value::Null);
}
#[cfg(unix)]
@@ -880,7 +898,7 @@ pub(crate) mod tests {
let log = l.strata_home().join("x.engine.log");
let exe = fake_strata(&l, &format!("rm -f '{}'; echo 'strata-q36: hybrid experts: 0 of 256 per layer in VRAM, bigger prompt chunks streamed (1.10 GiB staging); 7.1 s' >> '{}'; echo 'Strata is serving.'",
l.data.join("gone").display(), log.display()));
// The game PC in M0 (16303 MiB card, 1460 MiB held by other programs); the LLM without
// The game PC in M0 (16303 MiB card, `OTHERS` held by other programs); the LLM without
// experts held 9874 MiB.
let llm = Llm::with_exe(l.clone(), exe).with_gpu(card(&l, "GPU-test", 0));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
@@ -903,7 +921,27 @@ pub(crate) mod tests {
assert!(launches[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", launches[0]);
}
/// A fake card for `fake_placed`: 16303 MiB, 1460 MiB held by other programs (and by any
#[tokio::test]
async fn the_llm_log_tells_how_much_of_the_card_is_free_once_the_model_answers() {
crate::logs::install(crate::logs::LogHub::new(None));
let l = layout("llm-free");
let port = server_after_launch(&l, "list").await;
// The driver keeps some of the card for itself (`memory.reserved`): it is neither used nor free.
let inner = card(&l, "GPU-test", 0);
let reserved = 444;
let reader: GpuReader = Arc::new(move || inner().map(|g| Gpu { free_mib: g.free_mib - reserved, ..g }));
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(reader);
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
let free = 16303 - OTHERS - 9874 - reserved;
let lines = crate::logs::installed().unwrap()
.query(Some("llm"), crate::logs::Level::Info, Some("free"), None, 1000);
assert!(lines.iter().any(|x| x.text.contains(&format!("{free} MiB free of 16303 MiB"))),
"{:?}", lines.iter().map(|x| &x.text).collect::<Vec<_>>());
llm.stop().await.unwrap();
}
/// A fake card for `fake_placed`: 16303 MiB, `OTHERS` held by other programs (and by any
/// server still left over, `leftover` MiB, until a `launch --stop`), and 9874 MiB more while a
/// Strata server runs (the calls file shows the last launch and stop).
#[cfg(unix)]
@@ -913,8 +951,8 @@ pub(crate) mod tests {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let stopped = c.contains("launch --stop");
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
let used = 1460 + if running { 9874 } else if stopped { 0 } else { leftover };
Some(Gpu { id: id.into(), used_mib: used, total_mib: 16303 })
let used = OTHERS + if running { 9874 } else if stopped { 0 } else { leftover };
Some(gpu(id, used, 16303))
})
}
@@ -948,7 +986,7 @@ pub(crate) mod tests {
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
let r2 = reads.clone();
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: 1460, total_mib: 16303 })
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| gpu("GPU-test", OTHERS, 16303))
}));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
@@ -969,7 +1007,7 @@ pub(crate) mod tests {
#[cfg(unix)]
#[tokio::test]
async fn the_measurement_is_kept_per_requests_at_once_and_per_card() {
async fn the_measurement_is_kept_per_context_and_per_card() {
let l = layout("llm-place-keys");
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(card(&l, "GPU-a", 0));
let port = server_after_launch(&l, "list").await;
@@ -977,13 +1015,11 @@ pub(crate) mod tests {
llm.start(&one).await.unwrap();
assert_eq!(launches(&l).len(), 2, "measured once");
llm.stop().await.unwrap();
// A chat channel that answers everything: two requests at once, measured again.
// Another context: measured again.
std::fs::remove_file(l.data.join("calls")).unwrap();
let two = parse_settings(&json!({"llm": {"port": port}, "chatChannels": [{"id": "123456789012345678", "answer": "everything"}]})).unwrap();
assert_eq!(parallel(&two), 2);
llm.start(&two).await.unwrap();
assert_eq!(launches(&l).len(), 2, "measured for two at once");
assert!(launches(&l)[0].contains("--parallel 2"));
let short = parse_settings(&json!({"llm": {"port": port, "context": 16384}})).unwrap();
llm.start(&short).await.unwrap();
assert_eq!(launches(&l).len(), 2, "measured for the other context");
llm.stop().await.unwrap();
// Another card (same size): measured again.
std::fs::remove_file(l.data.join("calls")).unwrap();
@@ -991,7 +1027,7 @@ pub(crate) mod tests {
llm.start(&one).await.unwrap();
assert_eq!(launches(&l).len(), 2, "measured for the new card");
let probe: Value = serde_json::from_str(&std::fs::read_to_string(l.data.join("config").join("llm-probe.json")).unwrap()).unwrap();
assert_eq!((probe["32768/1"]["gpu"].clone(), probe["32768/2"]["gpu"].clone()), (json!("GPU-b"), json!("GPU-a")));
assert_eq!((probe["32768/1"]["gpu"].clone(), probe["16384/1"]["gpu"].clone()), (json!("GPU-b"), json!("GPU-a")));
llm.stop().await.unwrap();
}
@@ -1024,39 +1060,39 @@ pub(crate) mod tests {
llm.start(&s).await.unwrap();
assert_eq!(llm.stage_view(), json!({"stage": "normal", "gameRoomMib": null}));
std::fs::remove_file(l.data.join("calls")).unwrap();
assert_eq!(tokio::time::timeout(Duration::from_secs(30), llm.switch(&s, Stage::Gaming, false, true)).await.unwrap().unwrap(), Stage::Gaming);
assert_eq!(tokio::time::timeout(Duration::from_secs(30), llm.switch(&s, Stage::Gaming, 0, true)).await.unwrap().unwrap(), Stage::Gaming);
let g = launches(&l);
// Measured once at the game's context and one request, then the 2 GiB minimum (8 GB does not
// fit on 16 GB) with half the engine's 16 CPU threads.
assert_eq!(g.len(), 2, "{g:?}");
assert!(g[0].contains("--context 16384") && g[0].contains("--parallel 1") && g[0].contains("--set expert-vram=0"), "{}", g[0]);
assert!(g[1].ends_with("--set expert-vram=2 --set prefill=2048 --set pool-workers=8"), "{}", g[1]);
let room = 16303 - 1460 - 9874 - crate::placement::STT_MIB - crate::placement::MARGIN_MIB - 921;
let room = 16303 - OTHERS - 9874 - crate::placement::STT_MIB - crate::placement::TTS_MIB - crate::placement::MARGIN_MIB - 921;
assert_eq!(llm.stage_view(), json!({"stage": "gaming", "gameRoomMib": room}));
assert_eq!(llm.status().state, LlmState::Ready);
// Back to normal: the run's first reading of other programs, the normal placement again.
std::fs::remove_file(l.data.join("calls")).unwrap();
assert_eq!(llm.switch(&s, Stage::Normal, false, false).await.unwrap(), Stage::Normal);
assert_eq!(llm.switch(&s, Stage::Normal, 0, false).await.unwrap(), Stage::Normal);
let n = launches(&l);
assert_eq!(n.len(), 1, "{n:?}");
assert!(n[0].contains("--context 32768") && n[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", n[0]);
assert_eq!(llm.stage_view(), json!({"stage": "normal", "gameRoomMib": null}));
// Pause: the LLM stops.
assert_eq!(llm.switch(&s, Stage::Paused, false, true).await.unwrap(), Stage::Paused);
assert_eq!(llm.switch(&s, Stage::Paused, 0, true).await.unwrap(), Stage::Paused);
assert_eq!(llm.status().state, LlmState::Off);
assert_eq!(llm.stage_view()["stage"], json!("paused"));
}
/// A card whose other programs hold `others` MiB (changeable), plus the STT model while
/// `stt` is set, plus 9874 MiB while a Strata server runs.
/// A card whose other programs hold `others` MiB (changeable), plus the voice worker's models
/// (`worker` MiB), plus 9874 MiB while a Strata server runs.
#[cfg(unix)]
fn live_card(l: &Layout, others: Arc<std::sync::atomic::AtomicI64>, stt: bool) -> GpuReader {
fn live_card(l: &Layout, others: Arc<std::sync::atomic::AtomicI64>, worker: i64) -> GpuReader {
let calls = l.data.join("calls");
Arc::new(move || {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
let used = others.load(Ordering::SeqCst) + if stt { 1300 } else { 0 } + if running { 9874 } else { 0 };
Some(Gpu { id: "GPU-test".into(), used_mib: used, total_mib: 16303 })
let used = others.load(Ordering::SeqCst) + worker + if running { 9874 } else { 0 };
Some(gpu("GPU-test", used, 16303))
})
}
@@ -1065,25 +1101,48 @@ pub(crate) mod tests {
async fn the_normal_placement_reads_other_programs_afresh_and_a_game_is_not_counted_twice() {
let l = layout("llm-stage-fresh");
let port = server_after_launch(&l, "list").await;
let others = Arc::new(std::sync::atomic::AtomicI64::new(1460));
let others = Arc::new(std::sync::atomic::AtomicI64::new(OTHERS));
// At the start the STT worker has not loaded; at every switch it has.
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card(&l, others.clone(), false));
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card(&l, others.clone(), 0));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
llm.start(&s).await.unwrap();
let llm = llm.with_gpu(live_card(&l, others.clone(), true));
let stt = crate::placement::worker_models_mib(true, false);
let llm = llm.with_gpu(live_card(&l, others.clone(), stt));
// A game starts and holds 2500 MiB: the game stage counts other programs as before the game.
others.store(1460 + 2500, Ordering::SeqCst);
llm.switch(&s, Stage::Gaming, true, true).await.unwrap();
others.store(OTHERS + 2500, Ordering::SeqCst);
llm.switch(&s, Stage::Gaming, stt, true).await.unwrap();
assert!(launches(&l).last().unwrap().contains("--set expert-vram=2 "), "{:?}", launches(&l));
// The game ends but a video player now holds 1000 MiB more: the normal stage reads that.
others.store(1460 + 1000, Ordering::SeqCst);
llm.switch(&s, Stage::Normal, true, false).await.unwrap();
let expected = crate::placement::normal(16303, 1460 + 1000, &crate::placement::Probe { context: 32768, parallel: 1,
others.store(OTHERS + 1000, Ordering::SeqCst);
llm.switch(&s, Stage::Normal, stt, false).await.unwrap();
let expected = crate::placement::normal(16303, OTHERS + 1000, &crate::placement::Probe { context: 32768, parallel: 1,
gpu: "GPU-test".into(), model_mib: 9874, staging_gib: 1.1 }, 1024);
assert!(expected < 1.5);
assert!(launches(&l).last().unwrap().ends_with(&format!("--set expert-vram={expected} --set prefill=2048")), "{:?}", launches(&l));
}
#[cfg(unix)]
#[tokio::test]
async fn the_voice_models_loaded_after_the_start_change_nothing_on_the_way_back() {
// At the start neither the STT nor the TTS has loaded; at the switches both hold the card.
// With nothing else changed, the normal stage comes back with the placement it started with.
let l = layout("llm-stage-voice");
let port = server_after_launch(&l, "list").await;
let others = Arc::new(std::sync::atomic::AtomicI64::new(460));
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card(&l, others.clone(), 0));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
llm.start(&s).await.unwrap();
let first = launches(&l).last().unwrap().clone();
let voice = crate::placement::worker_models_mib(true, true);
let held = crate::placement::STT_MODEL_MIB + crate::placement::TTS_MIB;
let llm = llm.with_gpu(live_card(&l, others.clone(), held));
llm.switch(&s, Stage::Gaming, voice, true).await.unwrap();
llm.switch(&s, Stage::Normal, voice, false).await.unwrap();
let back = launches(&l).last().unwrap().clone();
assert!(!first.contains("--set expert-vram=0 "), "the test has room for experts: {first}");
assert_eq!(back, first);
}
/// `live_card` on a card of `total` MiB.
#[cfg(unix)]
fn live_card_of(l: &Layout, others: Arc<std::sync::atomic::AtomicI64>, total: i64) -> GpuReader {
@@ -1091,7 +1150,7 @@ pub(crate) mod tests {
Arc::new(move || {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
Some(Gpu { id: "GPU-test".into(), used_mib: others.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total_mib: total })
Some(gpu("GPU-test", others.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total))
})
}
@@ -1101,33 +1160,33 @@ pub(crate) mod tests {
// A game holds 2500 MiB already: it is in what the card holds, so no reserve on top.
let l = layout("llm-stage-game-at-start");
let port = server_after_launch(&l, "list").await;
let others = Arc::new(std::sync::atomic::AtomicI64::new(1460 + 2500));
let others = Arc::new(std::sync::atomic::AtomicI64::new(OTHERS + 2500));
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card_of(&l, others.clone(), 16303));
llm.game_at_start(true);
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
llm.start(&s).await.unwrap();
assert_eq!(llm.switch(&s, Stage::Gaming, false, true).await.unwrap(), Stage::Gaming, "not paused");
assert_eq!(llm.switch(&s, Stage::Gaming, 0, true).await.unwrap(), Stage::Gaming, "not paused");
let probe16 = crate::placement::Probe { context: 16384, parallel: 1, gpu: "GPU-test".into(), model_mib: 9874, staging_gib: 1.1 };
let expected = crate::placement::normal(16303, 1460 + 2500, &probe16, 1024);
let expected = crate::placement::normal(16303, OTHERS + 2500, &probe16, 1024);
assert!(launches(&l).last().unwrap().contains(&format!("--set expert-vram={expected} ")), "{:?}", launches(&l));
}
#[cfg(unix)]
#[tokio::test]
async fn the_game_stage_reads_the_card_again_and_counts_what_grew_since_the_normal_reading() {
// Normal reading: 1460 MiB. Then a program takes 4000 MiB and the game 4500 MiB: together
// Normal reading: OTHERS. Then a program takes 4000 MiB and the game 4500 MiB: together
// more than the 8 GB reserve, so the placement leaves that much, not just 8 GB.
let l = layout("llm-stage-grew");
let port = server_after_launch(&l, "list").await;
let others = Arc::new(std::sync::atomic::AtomicI64::new(1460));
let others = Arc::new(std::sync::atomic::AtomicI64::new(OTHERS));
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card_of(&l, others.clone(), 24576));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
llm.start(&s).await.unwrap();
others.store(1460 + 4000 + 4500, Ordering::SeqCst);
llm.switch(&s, Stage::Gaming, false, true).await.unwrap();
others.store(OTHERS + 4000 + 4500, Ordering::SeqCst);
llm.switch(&s, Stage::Gaming, 0, true).await.unwrap();
let probe16 = crate::placement::Probe { context: 16384, parallel: 1, gpu: "GPU-test".into(), model_mib: 9874, staging_gib: 1.1 };
let expected = crate::placement::game(24576, 1460, &probe16, 8500).expert_gib;
assert_ne!(expected, crate::placement::game(24576, 1460, &probe16, 8192).expert_gib, "the test tells them apart");
let expected = crate::placement::game(24576, OTHERS, &probe16, 8500).expert_gib;
assert_ne!(expected, crate::placement::game(24576, OTHERS, &probe16, 8192).expert_gib, "the test tells them apart");
assert!(launches(&l).last().unwrap().contains(&format!("--set expert-vram={expected} ")), "{:?}", launches(&l));
}
@@ -1144,9 +1203,9 @@ pub(crate) mod tests {
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
let r2 = reads.clone();
let llm = llm.with_gpu(Arc::new(move || {
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: 1460, total_mib: 16303 })
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| gpu("GPU-test", OTHERS, 16303))
}));
assert_eq!(llm.switch(&s, Stage::Gaming, false, true).await.unwrap(), Stage::Gaming);
assert_eq!(llm.switch(&s, Stage::Gaming, 0, true).await.unwrap(), Stage::Gaming);
assert!(launches(&l).last().unwrap().contains("--set expert-vram=0 "), "{:?}", launches(&l));
assert_eq!(llm.stage_view(), json!({"stage": "gaming", "gameRoomMib": null}));
}
@@ -1161,11 +1220,11 @@ pub(crate) mod tests {
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
Some(Gpu { id: "GPU-small".into(), used_mib: 1460 + if running { 9874 } else { 0 }, total_mib: 12288 })
Some(gpu("GPU-small", OTHERS + if running { 9874 } else { 0 }, 12288))
}));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
llm.start(&s).await.unwrap();
assert_eq!(llm.switch(&s, Stage::Gaming, false, true).await.unwrap(), Stage::Paused);
assert_eq!(llm.switch(&s, Stage::Gaming, 0, true).await.unwrap(), Stage::Paused);
assert_eq!(llm.status().state, LlmState::Off);
assert_eq!(llm.stage_view(), json!({"stage": "paused", "gameRoomMib": null}));
let c = std::fs::read_to_string(l.data.join("calls")).unwrap();

View File

@@ -470,8 +470,9 @@ impl WorkerClient {
}
/// Ask whether `text` was said to Jarvis (call mode `addressed`).
pub fn addressed(&self, id: &str, user: u64, text: &str) -> bool {
self.send_json(json!({"type": "addressed", "id": id, "user": user.to_string(), "text": text}))
pub fn addressed(&self, id: &str, user: u64, text: &str, people: usize, callers: usize) -> bool {
self.send_json(json!({"type": "addressed", "id": id, "user": user.to_string(), "text": text,
"people": people, "callers": callers}))
}
/// Tell the brain the LLM switches (voice_worker.spec.md, "LLM switches"): it starts no new
@@ -563,6 +564,14 @@ impl TtsBackend for WorkerClient {
fn tts_cancel(&self, id: &str) {
self.send_json(json!({"type": "tts_cancel", "id": id}));
}
fn tts_played(&self, id: &str, played: u32, cancelled: bool, last_ms: Option<u32>) {
let mut m = json!({"type": "tts_played", "id": id, "played": played, "cancelled": cancelled});
if let Some(ms) = last_ms {
m["last_ms"] = json!(ms);
}
self.send_json(m);
}
}
fn mark_disconnected(inner: &Inner) {

View File

@@ -30,7 +30,7 @@ class FakeBrain:
monitor 2 through the screen backend; a task answers "task: <text>"."""
def reply(self, user, text, history, chat=None, schedule=None, task=False, notes=None, notes_backend=None,
screen=None):
screen=None, trace=None):
if task:
return f"task: {text}" + (f" notes={'|'.join(notes)}" if notes else "")
if "지침" in text and notes_backend is not None:
@@ -45,7 +45,7 @@ class FakeBrain:
return f"scheduled {got.get('item', {}).get('id')} {got.get('error', '')}".strip()
return "첫 문장입니다. 둘째 문장입니다."
def addressed(self, user, text, names):
def addressed(self, user, text, names, people=None, callers=None):
return "켜" in text
def rank(self, candidates):

View File

@@ -367,12 +367,12 @@ async fn addressed_and_rank_questions_go_out_and_their_answers_come_back() {
let (c, mut rx) = client(&w.url(), json!({}));
c.start();
c.wait_ready(Duration::from_secs(2)).await.expect("ready");
assert!(c.addressed("q1", 42, "불 좀 꺼 줄래"));
assert!(c.addressed("q1", 42, "불 좀 꺼 줄래", 3, 1));
assert!(c.rank("k2", &[("7".into(), 42, "날씨".into()), ("8".into(), 43, "불 꺼".into())]));
until(|| w.got().len() >= 3).await;
let got: Vec<Value> = w.got()[1..].iter().map(|g| match g { Got::Json(v) => v.clone(), Got::Bin(_) => json!(null) }).collect();
assert_eq!(got, vec![
json!({"type": "addressed", "id": "q1", "user": "42", "text": "불 좀 꺼 줄래"}),
json!({"type": "addressed", "id": "q1", "user": "42", "text": "불 좀 꺼 줄래", "people": 3, "callers": 1}),
json!({"type": "rank", "id": "k2", "candidates": [{"id": "7", "user": "42", "text": "날씨"},
{"id": "8", "user": "43", "text": "불 꺼"}]}),
]);

View File

@@ -129,7 +129,7 @@ async fn audio_and_tts_round_trip_through_the_python_server() {
assert_eq!((end.id.as_str(), end.played, end.failed), (id.as_str(), 2, false));
// The two judgements come back under their ids.
assert!(c.addressed("q1", 42, "불 켜 줘"));
assert!(c.addressed("q1", 42, "불 켜 줘", 2, 2));
assert!(c.rank("k1", &[("1".into(), 42, "a".into()), ("2".into(), 43, "b".into())]));
let (mut addressed, mut ranked) = (None, None);
tokio::time::timeout(Duration::from_secs(5), async {

View File

@@ -385,7 +385,8 @@ const ACTIVITY = { "chat.send": "채팅 보내기", "chat.edit": "채팅 고치
"chat.delete": "채팅 지우기", "chat.pin": "고정", "chat.unpin": "고정 풀기",
"schedule.remind": "예약 알림 전달", "schedule.task": "정기 작업 전달", "collect.notify": "수집 소식 알림" };
/** What Jarvis did outside its answers (admin.spec.md, "Activity"): kind, target, who asked, outcome. */
function activityScreen() {
/** The 로그 screen's 한 일 view: what Jarvis did in chat channels and schedule deliveries. */
function activityView() {
const list = h("div");
const load = async () => {
try {
@@ -400,9 +401,9 @@ function activityScreen() {
} catch (err) { fail(err); }
};
load();
return h("div", {}, h("h1", {}, "한 일"), h("div", { class: "card" },
return h("div", { class: "card" },
h("div", { class: "hint" }, "자비스가 채팅 채널에서 한 동작과 예약 전달을 종류, 대상, 시킨 사람, 결과만 남깁니다. 내용은 남기지 않고 30일 뒤 지웁니다."),
h("div", { class: "field" }, h("button", { class: "btn", onclick: load }, "새로 고침")), list));
h("div", { class: "field" }, h("button", { class: "btn", onclick: load }, "새로 고침")), list);
}
// ---- conversations ------------------------------------------------------------------------------
@@ -449,7 +450,9 @@ function conversationScreen() {
const day = h("input", { type: "date" });
const find = h("input", { type: "text", placeholder: "찾기 (말, 답, 생각)", style: "max-width:220px" });
const live = h("input", { type: "checkbox", class: "switch", checked: true });
const open = new Set(); // traces the user opened stay open across refreshes
// Traces the user opened stay open across refreshes, and so does one seen while being made
// unless the user closed it.
const open = new Set(), shut = new Set();
// `page`: the live window and the older pages (talk.js). Each filter change starts a new epoch;
// answers from an earlier epoch, and live answers older than the newest one asked, are dropped.
let expandAll = false, sig = null, page = empty(), missing = false, sessionAt = null;
@@ -474,15 +477,20 @@ function conversationScreen() {
const where = h("span", { class: "small muted" }, timeOf(i.at), " · ", VIA[i.via] || i.via, i.channel ? " · " + channelName(i.channel) : "");
if (i.type === "thought") {
const tools = (i.steps || []).filter((s) => s.kind === "tool" && !s.skipped).length;
const summary = h("summary", {}, h("span", { class: "who" }, "자비스 생각"), " ",
h("span", { class: "small muted" }, `${i.who.name || i.who.id}의 말에 답하며`, tools ? ` · 도구 ${tools}번` : "",
i.pending ? " · 답을 만드는 중" : ` · ${(i.ms / 1000).toFixed(1)}초${i.answer ? "" : " · 답 없음"}`), " ", where);
if (i.pending && !shut.has(i.key)) open.add(i.key); // stays open with its answer once done
const d = h("details", { class: "msg thought", open: expandAll || open.has(i.key) },
h("summary", {}, h("span", { class: "who" }, "자비스 생각"), " ",
h("span", { class: "small muted" }, `${i.who.name || i.who.id}의 말에 답하며`, tools ? ` · 도구 ${tools}번` : "",
` · ${(i.ms / 1000).toFixed(1)}초`, i.answer ? "" : " · 답 없음"), " ", where),
summary,
h("div", { class: "steps" },
h("div", { class: "step-row" }, h("div", { class: "k" }, "받은 말"), h("div", {}, i.question)),
(i.steps || []).map(traceStep),
h("div", { class: "step-row" }, h("div", { class: "k" }, "답"), i.answer ? h("div", {}, i.answer) : h("span", { class: "badge bad" }, "답하지 못함"))));
d.addEventListener("toggle", () => (d.open ? open.add(i.key) : open.delete(i.key)));
h("div", { class: "step-row" }, h("div", { class: "k" }, "답"), i.pending ? h("div", {}, h("span", { class: "badge warn" }, "만드는 중"))
: i.answer ? h("div", {}, i.answer) : h("span", { class: "badge bad" }, "답하지 못함"))));
summary.addEventListener("click", () => {
if (d.open) { open.delete(i.key); shut.add(i.key); } else { open.add(i.key); shut.delete(i.key); }
});
return d;
}
const mine = i.type === "answered";
@@ -576,12 +584,26 @@ function conversationScreen() {
h("div", { class: "row" }, saidChip, answeredChip, thoughtChip, sessionChip),
h("div", { class: "row" }, via, person, channel, day, find),
h("div", { class: "row" }, h("label", { class: "row small muted" }, live, "실시간"),
h("button", { class: "btn", onclick: () => { expandAll = !expandAll; if (!expandAll) open.clear(); sig = null; load(); } }, "생각 모두 펼치기/접기"),
h("button", { class: "btn", onclick: () => { expandAll = !expandAll; open.clear(); shut.clear(); sig = null; load(); } }, "생각 모두 펼치기/접기"),
h("button", { class: "btn", onclick: () => { sig = null; load(); } }, "새로 고침"), summary)),
h("div", { class: "hint", style: "margin:8px 0" }, "사람이 한 말, 자비스의 답, 자비스가 답을 만든 과정(참고한 기록, 고른 도구, 계획, 도구 호출과 결과, 모델 생각)을 시간순으로 보여 줍니다. 모델 생각은 생각 모드가 켜진 답에만 있습니다. 생각 기록은 30일 동안 둡니다. 실시간으로 다시 읽는 것은 최근 2,000개까지이고, 그보다 앞서 불러온 기록의 수정과 삭제는 화면을 다시 열면 보입니다."),
h("div", { class: "hint", style: "margin:8px 0" }, "사람이 한 말, 자비스의 답, 자비스가 답을 만든 과정(참고한 기록, 고른 도구, 계획, 도구 호출과 결과, 모델 생각)을 시간순으로 보여 줍니다. 자비스가 말을 알아들은 순간 생각이 나타나고, 답을 만드는 동안 단계가 하나씩 채워집니다. 모델 생각은 생각 모드가 켜진 답에만 있습니다. 생각 기록은 30일 동안 둡니다. 실시간으로 다시 읽는 것은 최근 2,000개까지이고, 그보다 앞서 불러온 기록의 수정과 삭제는 화면을 다시 열면 보입니다."),
h("div", { class: "row", style: "justify-content:center" }, more), box);
load();
return { view, tick: () => live.checked && load() };
// Twice a second the store's stamp (cheap): when something was said, a question arrived or an
// answer moved on, the window is read again at once, so a trace shows from its question on and
// fills in as it goes. Every 3 seconds it is read anyway (an edited message changes no stamp).
let ticks = 0, stamp = null, asking = false;
const tick = async () => {
if (!live.checked) return;
if (++ticks % 6 === 0) return load();
if (asking) return;
asking = true;
try {
const s = (await api("history_stamp")).stamp;
if (s !== stamp) { stamp = s; load(); }
} catch (e) { /* the next tick tries again */ } finally { asking = false; }
};
return { view, tick };
}
async function settingsScreen(screen, title, extra) {
@@ -663,6 +685,28 @@ async function tokenCard() {
} }, "확인 후 저장")), out);
}
/** The KMA API hub key for the weather inside Korea: typed here, stored encrypted, only shown masked. */
async function kmaKeyCard() {
const label = h("span");
const input = h("input", { type: "password", placeholder: "기상청 API허브 인증키", autocomplete: "off" });
const clear = h("button", { class: "btn" }, "키 지우기");
const show = (st) => { label.textContent = st.set ? st.masked : "저장된 키 없음"; clear.hidden = !st.set; };
const put = async (value) => {
try {
const r = await api("kma_key_set", { value });
input.value = "";
show(r);
toast(r.restartNeeded ? "저장했습니다. 다시 시작하면 적용됩니다." : "저장했습니다.");
} catch (e) { fail(e); }
};
clear.onclick = () => put("");
show(await api("kma_key_status"));
return h("div", { class: "card" }, h("h2", {}, "기상청 날씨"),
h("p", { class: "small muted" }, "기상청 API허브(apihub.kma.go.kr) 인증키를 넣으면 한국 안의 현재 날씨를 기상청에서 받습니다. 키가 없거나 기상청이 답하지 않으면 open-meteo, met.no 순서로 받습니다. API허브에서 단기예보의 초단기실황조회와 초단기예보조회를 활용신청해야 합니다."),
h("div", { class: "row" }, label, input,
h("button", { class: "btn", onclick: () => input.value.trim() ? put(input.value) : toast("키를 입력해 주세요", true) }, "저장"), clear));
}
/** The virtual monitor's driver for "봇용 크롬창 화면공유 선택 표시" (Windows only). */
async function displayDriverCard() {
const st = await api("display_driver_status");
@@ -801,7 +845,7 @@ function logsScreen() {
find.addEventListener("input", () => { clearTimeout(find.t); find.t = setTimeout(reload, 300); });
const today = new Date().toISOString().slice(0, 10);
const from = h("input", { type: "date", value: today }), to = h("input", { type: "date", value: today });
const view = h("div", {}, h("h1", {}, "로그"),
const logs = h("div", {},
h("div", { class: "row", style: "margin-bottom:10px" }, tab, level, find, h("label", { class: "row small muted" }, pause, "일시정지"),
h("button", { class: "btn", onclick: async () => {
try {
@@ -810,7 +854,15 @@ function logsScreen() {
await navigator.clipboard.writeText(r.report);
toast(r.tooLongForDiscord ? "복사했습니다. 디스코드 메시지 한도(2,000자)를 넘으니 파일로 내보내세요." : "최근 오류를 복사했습니다.");
} catch (e) { fail(e); }
} }, "최근 오류 복사")),
} }, "최근 오류 복사"),
h("button", { class: "btn", onclick: async () => {
try {
const text = [...box.children].map((l) => l.textContent).join("\n");
if (!text) return toast("화면에 보이는 로그가 없습니다");
const r = await api("logs_save_text", { text });
toast(`${r.path}에 저장했습니다`);
} catch (e) { fail(e); }
} }, "보이는 글 txt로 저장")),
box,
h("div", { class: "row", style: "margin-top:10px" }, h("span", { class: "muted" }, "내보내기"), from, "~", to,
h("button", { class: "btn", onclick: async () => {
@@ -819,14 +871,70 @@ function logsScreen() {
toast(`${r.path}에 저장했습니다${r.droppedLines ? ` (10MB를 넘어 오래된 ${r.droppedLines}줄은 뺐습니다)` : ""}`);
} catch (e) { fail(e); }
} }, "zip으로 저장")));
// 로그, 대화 (its export), 목소리 (the recordings) and 한 일 (the activity) on one screen; each view
// but 로그 is made again each time it is opened.
const body = h("div");
const switches = [["로그", () => logs], ["대화", talkExportView], ["목소리", voiceView], ["한 일", activityView]].map(([label, make]) => {
const b = h("button", { class: "btn", onclick: () => {
for (const x of switches) x.classList.toggle("primary", x === b);
body.replaceChildren(make());
} }, label);
return b;
});
const view = h("div", {}, h("h1", {}, "로그"), h("div", { class: "row", style: "margin-bottom:12px" }, switches), body);
switches[0].click();
reload();
return { view, tick: poll };
return { view, tick: () => logs.isConnected && poll() };
}
/** Two date boxes for a range of days, today by default (local time). */
function dayRange() {
const today = dayOf(Date.now() / 1000);
return [h("input", { type: "date", value: today }), h("input", { type: "date", value: today })];
}
/** The 로그 screen's 대화 view: the 대화 screen's content of some days, saved as a text file. */
function talkExportView() {
const [from, to] = dayRange();
return h("div", { class: "card" },
h("div", { class: "hint" }, "대화 화면의 내용을 날짜별로 txt 파일에 저장합니다. 사람이 한 말, 자비스 생각(참고한 기록, 고른 도구, 계획, 도구 호출과 가져온 자료), 자비스 답이 모두 들어갑니다. 파일은 데이터 폴더의 logs 폴더에 저장되고 이 PC에만 남습니다."),
h("div", { class: "row" }, from, "~", to, h("button", { class: "btn primary", onclick: async () => {
try {
const r = await api("history_export", { from: from.value, to: to.value });
toast(r.items ? `${r.items}개를 ${r.path}에 저장했습니다` : `그 날짜에는 기록이 없습니다 (${r.path})`);
} catch (e) { fail(e); }
} }, "txt로 저장")));
}
/** The 로그 screen's 목소리 view: whether recordings are kept, and the recordings of some days as a zip. */
function voiceView() {
const on = h("input", { type: "checkbox", class: "switch" });
api("settings_view").then((v) => { on.checked = !!v["voice.record"]; }).catch(fail);
on.addEventListener("change", async () => {
try {
const r = await api("settings_apply", { changes: { "voice.record": on.checked } });
toast(r.restartNeeded ? "저장했습니다. 봇을 다시 시작하면 적용됩니다." : "저장했습니다.");
} catch (e) { on.checked = !on.checked; fail(e); }
});
const [from, to] = dayRange();
return h("div", { class: "card" },
h("div", { class: "hint" }, "켜 두면 음성 채널에서 받아 적은 말 하나하나와 자비스가 말한 답을 소리 파일(wav)로 저장합니다. 같은 채널의 다른 사람 목소리도 저장되니 필요할 때만 켜 주세요. 데이터 폴더의 recordings 폴더에 7일 동안 남고 그 뒤 지웁니다. 봇을 다음에 시작할 때부터 적용됩니다."),
h("label", { class: "row", style: "margin-bottom:12px" }, on, "목소리 녹음 저장"),
h("div", { class: "row" }, from, "~", to, h("button", { class: "btn primary", onclick: async () => {
try {
const r = await api("voice_export", { from: from.value, to: to.value });
const mb = (r.bytes / 1e6).toFixed(1);
toast(r.files ? `소리 파일 ${r.files}개(${mb}MB)를 ${r.path}에 저장했습니다${r.bytes > 10e6 ? ". 디스코드 파일 한도(10MB)를 넘습니다" : ""}`
: `그 날짜에는 저장된 목소리가 없습니다 (${r.path})`);
} catch (e) { fail(e); }
} }, "zip으로 저장")),
h("div", { class: "small muted", style: "margin-top:8px" }, "zip 안의 index.txt에 파일마다 시간, 누가 말했는지, 받아 적은 글(자비스는 말한 글)이 적혀 있습니다."));
}
// ---- update -------------------------------------------------------------------------------------
/** The update screen: the version here, the newest release, its download and install. */
function updateScreen() {
/** The 앱 screen's update card: the version here, the newest release, its download and install. */
function updateCard() {
const body = h("div");
let key = null;
async function tick() {
@@ -844,7 +952,7 @@ function updateScreen() {
const line = { idle: "아직 확인하지 않았습니다", checking: "확인하는 중…", upToDate: "최신 버전입니다",
available: `새 버전 ${u.latest?.version}이 있습니다`, downloading: `받는 중${pct != null ? ` ${pct}%` : ""}`,
ready: `${u.latest?.version}을 설치할 준비가 됐습니다`, failed: u.error ?? "실패했습니다" }[u.phase] ?? u.phase;
body.replaceChildren(h("div", { class: "card" },
body.replaceChildren(h("div", { class: "card" }, h("h2", {}, "업데이트"),
h("div", { class: "kv" }, h("span", { class: "k" }, "현재 버전"), h("span", {}, u.current),
h("span", { class: "k" }, "상태"), h("span", { style: u.phase === "failed" ? "color:var(--bad)" : "" }, line)),
h("div", { class: "row", style: "margin-top:12px" },
@@ -856,15 +964,15 @@ function updateScreen() {
h("pre", { class: "small", style: "white-space:pre-wrap" }, u.latest.notes)) : "");
}
tick();
return { view: h("div", {}, h("h1", {}, "업데이트"), body), tick };
return { view: body, tick };
}
/** Look for a new version once when the app opens; the nav button says so when there is one. */
async function checkUpdate() {
try {
const u = await api("update_check");
if (u.latest) { $("#nav-update").textContent = "업데이트 (새 버전)"; toast(`새 버전 ${u.latest.version}이 있습니다. 업데이트 화면에서 설치할 수 있습니다.`); }
} catch (e) { /* offline: the update screen shows why */ }
if (u.latest) { $("#nav-app").textContent = "앱 (새 버전)"; toast(`새 버전 ${u.latest.version}이 있습니다. 앱 화면에서 설치할 수 있습니다.`); }
} catch (e) { /* offline: the 앱 screen shows why */ }
}
// ---- navigation ---------------------------------------------------------------------------------
@@ -900,19 +1008,20 @@ async function show(name) {
try {
if (name === "dashboard") { screen.replaceChildren(); await refresh(); ticker = setInterval(refresh, 1000); return; }
if (name === "logs") { const l = logsScreen(); screen.replaceChildren(l.view); ticker = setInterval(l.tick, 1000); return; }
if (name === "activity") { screen.replaceChildren(activityScreen()); return; }
if (name === "conversations") { const c = conversationScreen(); screen.replaceChildren(c.view); ticker = setInterval(c.tick, 3000); return; }
if (name === "conversations") { const c = conversationScreen(); screen.replaceChildren(c.view); ticker = setInterval(c.tick, 500); return; }
if (name === "notes") { screen.replaceChildren(notesScreen()); return; }
if (name === "update") { const u = updateScreen(); screen.replaceChildren(u.view); ticker = setInterval(u.tick, 1000); return; }
const titles = { bot: "봇", channels: "채널·호출", chat: "채팅 채널", memory: "대화 기억", model: "모델", features: "기능",
collect: "백그라운드 수집", game: "게임 모드", schedule: "예약", pcControl: "PC 제어", app: "앱" };
const extra = name === "bot" ? await tokenCard() : name === "features" ? await displayDriverCard()
: name === "app" ? h("div", { class: "hint", style: "margin-bottom:12px" }, "트레이로 최소화하면 창만 숨고 봇과 서비스는 계속 돕니다. 작업 표시줄 오른쪽의 Jarvis 아이콘을 누르면 창이 다시 열리고, 아이콘의 오른쪽 클릭 메뉴에서 완전 종료할 수 있습니다. 저장하면 바로 적용됩니다.") : null;
const update = name === "app" ? updateCard() : null;
const extra = name === "bot" ? await tokenCard() : name === "features" ? h("div", {}, await kmaKeyCard(), await displayDriverCard())
: name === "app" ? h("div", {}, update.view, h("div", { class: "hint", style: "margin:12px 0" }, "트레이로 최소화하면 창만 숨고 봇과 서비스는 계속 돕니다. 작업 표시줄 오른쪽의 Jarvis 아이콘을 누르면 창이 다시 열리고, 아이콘의 오른쪽 클릭 메뉴에서 완전 종료할 수 있습니다. 저장하면 바로 적용됩니다.")) : null;
const view = await settingsScreen(name, titles[name], extra);
if (name === "collect") view.append(await collectResults());
if (name === "schedule") view.append(await scheduleList());
if (name === "pcControl") view.append(await permissionsCard());
if (my === gen) screen.replaceChildren(view);
if (my !== gen) return;
screen.replaceChildren(view);
if (update) ticker = setInterval(update.tick, 1000);
} catch (e) { fail(e); }
}

View File

@@ -23,11 +23,9 @@
<button data-screen="schedule" class="nav">예약</button>
<button data-screen="pcControl" class="nav">PC 제어</button>
<button data-screen="notes" class="nav">사람별 지침</button>
<button data-screen="activity" class="nav">한 일</button>
<button data-screen="game" class="nav">게임 모드</button>
<button data-screen="app" class="nav">앱</button>
<button data-screen="logs" class="nav">로그</button>
<button data-screen="update" class="nav" id="nav-update">업데이트</button>
<button data-screen="app" class="nav" id="nav-app">앱</button>
<div class="side-foot" id="version"></div>
</nav>
<main id="screen" class="screen"></main>

View File

@@ -0,0 +1,6 @@
{
"git": {
"sha1": "957e17fcbc687cd88979c90b76bdf79017803b09"
},
"path_in_vcs": ""
}

376
app/vendor/songbird/Cargo.toml vendored Normal file
View File

@@ -0,0 +1,376 @@
# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
#
# When uploading crates to the registry Cargo will automatically
# "normalize" Cargo.toml files for maximal compatibility
# with all versions of Cargo and also rewrite `path` dependencies
# to registry (e.g., crates.io) dependencies.
#
# If you are reading this file be aware that the original Cargo.toml
# will likely look very different (and much more reasonable).
# See Cargo.toml.orig for the original contents.
[package]
edition = "2021"
rust-version = "1.83.0"
name = "songbird"
version = "0.6.1"
authors = ["Kyle Simpson <kyleandrew.simpson@gmail.com>"]
build = "build.rs"
include = [
"src/**/*.rs",
"Cargo.toml",
"build.rs",
]
autolib = false
autobins = false
autoexamples = false
autotests = false
autobenches = false
description = "An async Rust library for the Discord voice API."
homepage = "https://github.com/serenity-rs/songbird"
documentation = "https://docs.rs/songbird"
readme = "README.md"
keywords = [
"discord",
"api",
"rtp",
"audio",
]
license = "ISC"
repository = "https://github.com/serenity-rs/songbird.git"
[package.metadata.docs.rs]
features = ["full-doc"]
rustdoc-args = [
"--cfg",
"docsrs",
]
[features]
builtin-queue = []
default = [
"driver",
"gateway",
"rustls",
"serenity",
"tungstenite",
]
driver = [
"dep:aead",
"dep:aes-gcm",
"dep:async-trait",
"dep:opus2",
"dep:byteorder",
"dep:bytes",
"dep:chacha20poly1305",
"dep:crypto-common",
"dep:davey",
"dep:discortp",
"dep:flume",
"dep:nohash-hasher",
"dep:parking_lot",
"dep:rand",
"dep:reqwest",
"dep:ringbuf",
"dep:rubato",
"dep:rusty_pool",
"dep:serde-aux",
"dep:serenity-voice-model",
"dep:socket2",
"dep:stream_lib",
"dep:streamcatcher",
"dep:symphonia",
"dep:symphonia-core",
"dep:tokio",
"dep:tokio-util",
"dep:typenum",
"dep:url",
"dep:uuid",
"tokio?/fs",
"tokio?/io-util",
"tokio?/macros",
"tokio?/net",
"tokio?/process",
"tokio?/rt",
"tokio?/sync",
"tokio?/time",
]
full-doc = [
"default",
"tungstenite",
"twilight",
"builtin-queue",
"receive",
]
gateway = [
"dep:async-trait",
"dep:dashmap",
"dep:flume",
"dep:parking_lot",
"dep:tokio",
"tokio?/sync",
"tokio?/time",
]
internals = ["dep:byteorder"]
native = [
"reqwest?/native-tls",
"serenity?/native_tls_backend",
"stream_lib?/native-tls",
"tokio-tungstenite?/native-tls",
"tokio-websockets?/native-tls",
"twilight-gateway?/native-tls",
]
receive = [
"dep:bytes",
"discortp?/demux",
"discortp?/rtcp",
]
rustls = [
"reqwest?/rustls-tls",
"serenity?/rustls_backend",
"stream_lib?/rustls-tls",
"tokio-tungstenite?/rustls-tls-webpki-roots",
"tokio-websockets?/ring",
"tokio-websockets?/rustls-native-roots",
"twilight-gateway?/rustls-native-roots",
]
tungstenite = ["dep:tokio-tungstenite"]
twilight = [
"dep:twilight-gateway",
"dep:twilight-model",
]
tws = ["dep:tokio-websockets"]
[lib]
name = "songbird"
path = "src/lib.rs"
bench = false
[dependencies.aead]
version = "0.5.2"
optional = true
[dependencies.aes-gcm]
version = "0.10.3"
optional = true
[dependencies.async-trait]
version = "0.1"
optional = true
[dependencies.byteorder]
version = "1"
optional = true
[dependencies.bytes]
version = "1"
optional = true
[dependencies.chacha20poly1305]
version = "0.10.1"
optional = true
[dependencies.crypto-common]
version = "0.1"
features = ["std"]
optional = true
[dependencies.dashmap]
version = "6.1.0"
optional = true
[dependencies.davey]
version = "0.1.2"
optional = true
[dependencies.derivative]
version = "2"
[dependencies.discortp]
version = "0.6"
features = [
"discord",
"pnet",
"rtp",
]
optional = true
default-features = false
[dependencies.either]
version = "1.9.0"
[dependencies.enum-map]
version = "2.7.3"
[dependencies.flume]
version = "0.11"
optional = true
[dependencies.futures]
version = "0.3"
[dependencies.nohash-hasher]
version = "0.2.0"
optional = true
[dependencies.nonmax]
version = "0.5.5"
[dependencies.opus2]
version = "0.4.0"
optional = true
[dependencies.parking_lot]
version = "0.12"
optional = true
[dependencies.pin-project]
version = "1"
[dependencies.rand]
version = "0.9"
optional = true
[dependencies.reqwest]
version = "0.12.2"
features = ["stream"]
optional = true
default-features = false
[dependencies.ringbuf]
version = "0.4"
optional = true
[dependencies.rubato]
version = "0.16"
optional = true
[dependencies.rusty_pool]
version = "0.7"
optional = true
[dependencies.serde]
version = "1"
features = ["derive"]
[dependencies.serde-aux]
version = "4"
optional = true
[dependencies.serde_json]
version = "1"
[dependencies.serenity]
version = "0.12.0"
features = [
"voice",
"gateway",
]
optional = true
default-features = false
[dependencies.serenity-voice-model]
version = "0.3"
optional = true
[dependencies.socket2]
version = "0.5"
optional = true
[dependencies.stream_lib]
version = "0.5.2"
optional = true
default-features = false
[dependencies.streamcatcher]
version = "1"
optional = true
[dependencies.symphonia]
version = "0.5.2"
optional = true
default-features = false
[dependencies.symphonia-core]
version = "0.5.2"
optional = true
[dependencies.tokio]
version = "1.0"
optional = true
default-features = false
[dependencies.tokio-tungstenite]
version = "0.26"
features = ["url"]
optional = true
[dependencies.tokio-util]
version = "0.7"
features = ["io"]
optional = true
[dependencies.tokio-websockets]
version = "0.11"
features = [
"client",
"fastrand",
"sha1_smol",
"simd",
]
optional = true
[dependencies.tracing]
version = "0.1"
features = ["log"]
[dependencies.tracing-futures]
version = "0.2"
[dependencies.twilight-gateway]
version = ">= 0.16.0, <= 0.17"
optional = true
default-features = false
[dependencies.twilight-model]
version = ">= 0.16.0, <= 0.17"
optional = true
default-features = false
[dependencies.typenum]
version = "1.17.0"
optional = true
[dependencies.url]
version = "2"
optional = true
[dependencies.uuid]
version = "1"
features = ["v4"]
optional = true
[dev-dependencies.byteorder]
version = "1"
[dev-dependencies.criterion]
version = "0.5"
[dev-dependencies.ntest]
version = "0.9"
[dev-dependencies.symphonia]
version = "0.5.2"
features = [
"aac",
"isomp4",
"mp3",
]
[dev-dependencies.tokio]
version = "1"
features = [
"rt",
"rt-multi-thread",
"test-util",
]

183
app/vendor/songbird/Cargo.toml.orig generated vendored Normal file
View File

@@ -0,0 +1,183 @@
[package]
authors = ["Kyle Simpson <kyleandrew.simpson@gmail.com>"]
description = "An async Rust library for the Discord voice API."
documentation = "https://docs.rs/songbird"
edition = "2021"
homepage = "https://github.com/serenity-rs/songbird"
include = ["src/**/*.rs", "Cargo.toml", "build.rs"]
keywords = ["discord", "api", "rtp", "audio"]
license = "ISC"
name = "songbird"
readme = "README.md"
repository = "https://github.com/serenity-rs/songbird.git"
rust-version = "1.83.0"
version = "0.6.1"
[dependencies]
aead = { optional = true, version = "0.5.2" }
aes-gcm = { optional = true, version = "0.10.3" }
async-trait = { optional = true, version = "0.1" }
opus2 = { optional = true, version = "0.4.0" }
byteorder = { optional = true, version = "1" }
bytes = { optional = true, version = "1" }
chacha20poly1305 = { optional = true, version = "0.10.1" }
crypto-common = { optional = true, features = ["std"], version = "0.1" }
dashmap = { optional = true, version = "6.1.0" }
davey = { optional = true, version = "0.1.2" }
derivative = "2"
discortp = { default-features = false, features = [
"discord",
"pnet",
"rtp",
], optional = true, version = "0.6" }
either = "1.9.0"
flume = { optional = true, version = "0.11" }
futures = "0.3"
nohash-hasher = { optional = true, version = "0.2.0" }
parking_lot = { optional = true, version = "0.12" }
pin-project = "1"
rand = { optional = true, version = "0.9" }
reqwest = { default-features = false, features = [
"stream",
], optional = true, version = "0.12.2" }
ringbuf = { optional = true, version = "0.4" }
rubato = { optional = true, version = "0.16" }
rusty_pool = { optional = true, version = "0.7" }
serde = { version = "1", features = ["derive"] }
serde-aux = { optional = true, version = "4" }
serde_json = "1"
serenity = { default-features = false, optional = true, version = "0.12.0", features = [
"voice",
"gateway",
] }
serenity-voice-model = { optional = true, version = "0.3" }
socket2 = { optional = true, version = "0.5" }
streamcatcher = { optional = true, version = "1" }
stream_lib = { default-features = false, optional = true, version = "0.5.2" }
symphonia = { default-features = false, optional = true, version = "0.5.2" }
symphonia-core = { optional = true, version = "0.5.2" }
tokio = { default-features = false, optional = true, version = "1.0" }
tokio-tungstenite = { optional = true, version = "0.26", features = ["url"] }
tokio-websockets = { optional = true, version = "0.11", features = [
"client",
"fastrand",
"sha1_smol",
"simd",
] }
tokio-util = { features = ["io"], optional = true, version = "0.7" }
tracing = { version = "0.1", features = ["log"] }
tracing-futures = "0.2"
twilight-gateway = { default-features = false, optional = true, version = ">= 0.16.0, <= 0.17" }
twilight-model = { default-features = false, optional = true, version = ">= 0.16.0, <= 0.17" }
typenum = { optional = true, version = "1.17.0" }
url = { optional = true, version = "2" }
uuid = { features = ["v4"], optional = true, version = "1" }
nonmax = "0.5.5"
enum-map = "2.7.3"
[dev-dependencies]
byteorder = "1"
criterion = "0.5"
ntest = "0.9"
symphonia = { version = "0.5.2", features = ["aac", "isomp4", "mp3"] }
tokio = { version = "1", features = ["rt", "rt-multi-thread", "test-util"] }
[features]
# Core features
default = ["driver", "gateway", "rustls", "serenity", "tungstenite"]
gateway = [
"dep:async-trait",
"dep:dashmap",
"dep:flume",
"dep:parking_lot",
"dep:tokio",
"tokio?/sync",
"tokio?/time",
]
driver = [
"dep:aead",
"dep:aes-gcm",
"dep:async-trait",
"dep:opus2",
"dep:byteorder",
"dep:bytes",
"dep:chacha20poly1305",
"dep:crypto-common",
"dep:davey",
"dep:discortp",
"dep:flume",
"dep:nohash-hasher",
"dep:parking_lot",
"dep:rand",
"dep:reqwest",
"dep:ringbuf",
"dep:rubato",
"dep:rusty_pool",
"dep:serde-aux",
"dep:serenity-voice-model",
"dep:socket2",
"dep:stream_lib",
"dep:streamcatcher",
"dep:symphonia",
"dep:symphonia-core",
"dep:tokio",
"dep:tokio-util",
"dep:typenum",
"dep:url",
"dep:uuid",
"tokio?/fs",
"tokio?/io-util",
"tokio?/macros",
"tokio?/net",
"tokio?/process",
"tokio?/rt",
"tokio?/sync",
"tokio?/time",
]
rustls = [
"reqwest?/rustls-tls",
"serenity?/rustls_backend",
"stream_lib?/rustls-tls",
"tokio-tungstenite?/rustls-tls-webpki-roots",
"tokio-websockets?/ring",
"tokio-websockets?/rustls-native-roots",
"twilight-gateway?/rustls-native-roots",
]
native = [
"reqwest?/native-tls",
"serenity?/native_tls_backend",
"stream_lib?/native-tls",
"tokio-tungstenite?/native-tls",
"tokio-websockets?/native-tls",
"twilight-gateway?/native-tls",
]
tungstenite = ["dep:tokio-tungstenite"]
tws = ["dep:tokio-websockets"]
twilight = ["dep:twilight-gateway", "dep:twilight-model"]
# Behaviour altering features.
builtin-queue = []
receive = ["dep:bytes", "discortp?/demux", "discortp?/rtcp"]
# Used for docgen/testing/benchmarking.
full-doc = ["default", "tungstenite", "twilight", "builtin-queue", "receive"]
internals = ["dep:byteorder"]
[lib]
bench = false
[[bench]]
name = "base-mixing"
path = "benches/base-mixing.rs"
required-features = ["internals"]
harness = false
[[bench]]
name = "mixing-task"
path = "benches/mixing-task.rs"
required-features = ["internals"]
harness = false
[package.metadata.docs.rs]
features = ["full-doc"]
rustdoc-args = ["--cfg", "docsrs"]

14
app/vendor/songbird/JARVIS-PATCH.md vendored Normal file
View File

@@ -0,0 +1,14 @@
# songbird 0.6.1, patched for Jarvis
This is songbird 0.6.1 from crates.io (https://github.com/serenity-rs/songbird, ISC licence, see its
README), used through `[patch.crates-io]` in `app/Cargo.toml` and `app/shell/Cargo.toml`.
The one change: RTP packets of an SSRC that Discord has not named yet are held for up to a second
instead of dropped, and processed once its Speaking op names it (`src/driver/tasks/udp_rx/held.rs`,
`udp_rx/mod.rs` `process_udp_message`; `ws.rs` sends the Speaking event before naming the SSRC so
handlers see the name before the held packets). Discord's Speaking op comes after a speaker's first
packets, and unpatched songbird dropped DAVE packets it could not yet decrypt, so the first ~100 ms
of the first thing each person said after a join were lost (live 2026-11-18: "자비스 서울 날씨 어때"
arrived as "아비 서울날씨 어때").
Drop this copy when an upstream release does the same.

97
app/vendor/songbird/README.md vendored Normal file
View File

@@ -0,0 +1,97 @@
[![crates.io badge]][crates.io] [![build badge]][build] [![docs badge]][docs] [![current docs badge]][current docs] [![next docs badge]][next docs] [![msrv badge]][msrv] [![license badge]][license] [![guild badge]][guild]
# Songbird
![](songbird.png)
Songbird is an async, cross-library compatible voice system for Discord, written in Rust.
The library offers:
* A standalone gateway frontend compatible with [serenity] and [twilight] using the
`"gateway"` and `"[serenity/twilight]"` plus `"[rustls/native]"` features. You can even run
driverless, to help manage your [lavalink] sessions.
* A standalone driver for voice calls, via the `"driver"` feature. If you can create
a `ConnectionInfo` using any other gateway, or language for your bot, then you
can run the songbird voice driver.
* Voice receive and RT(C)P packet handling via the `"receive"` feature.
* And, by default, a fully featured voice system featuring events, queues,
seeking on compatible streams, shared multithreaded audio stream caches,
and direct Opus data passthrough from DCA files.
## Intents
Songbird's gateway functionality requires you to specify the `GUILD_VOICE_STATES` intent.
## Codec support
Songbird supports all [codecs and formats provided by Symphonia] (pure-Rust), with Opus support
provided by [opus2] (an FFI wrapper for libopus).
**By default, *Songbird will not request any codecs from Symphonia*.** To change this, in your own
project you will need to depend on Symphonia as well.
```toml
# Including songbird alone gives you support for Opus via the DCA file format.
[dependencies.songbird]
version = "0.5"
features = ["builtin-queue"]
# To get additional codecs, you *must* add Symphonia yourself.
# This includes the default formats (MKV/WebM, Ogg, Wave) and codecs (FLAC, PCM, Vorbis)...
[dependencies.symphonia]
version = "0.5"
features = ["aac", "mp3", "isomp4", "alac"] # ...as well as any extras you need!
```
## Dependencies
Songbird needs a few system dependencies before you can use it.
- Opus - Audio codec that Discord uses.
[opus2] will use installed libopus binaries if available via pkgconf on Linux/MacOS, otherwise you will need to install cmake to build opus from source.
This is always the case on Windows.
For Unix systems, you can install the library with `apt install libopus-dev` on Ubuntu or `pacman -S opus` on Arch Linux.
If you do not have it installed it will be built for you. However, you will need a C compiler and the GNU autotools installed.
Again, these can be installed with `apt install build-essential autoconf automake libtool m4` on Ubuntu or `pacman -S base-devel` on Arch Linux.
This is a required dependency. Songbird cannot work without it.
- yt-dlp / youtube-dl / (similar forks) - Audio/Video download tool.
yt-dlp can be installed [according to the installation instructions on the main repo].
You can install youtube-dl with Python's package manager, pip, which we recommend for youtube-dl. You can do it with the command `pip install youtube_dl`.
Alternatively, you can install it with your system's package manager, `apt install youtube-dl` on Ubuntu or `pacman -S youtube-dl` on Arch Linux.
This is an optional dependency for users, but is required as a dev-dependency. It allows Songbird to download audio/video sources from the Internet from a variety of webpages, which it will convert to the Opus audio format Discord uses.
## Examples
Full examples showing various types of functionality and integrations can be found in [this crate's examples directory].
## Contributing
If you want to help out or file an issue, please look over [our contributor guidelines]!
## Attribution
Songbird's logo is based upon the copyright-free image ["Black-Capped Chickadee"] by George Gorgas White.
[serenity]: https://github.com/serenity-rs/serenity
[twilight]: https://github.com/twilight-rs/twilight
["Black-Capped Chickadee"]: https://www.oldbookillustrations.com/illustrations/black-capped-chickadee/
[lavalink]: https://github.com/freyacodes/Lavalink
[this crate's examples directory]: https://github.com/serenity-rs/songbird/tree/current/examples
[our contributor guidelines]: CONTRIBUTING.md
[codecs and formats provided by Symphonia]: https://github.com/pdeljanov/Symphonia#formats-demuxers
[opus2]: https://github.com/cijiugechu/opus2
[according to the installation instructions on the main repo]: https://github.com/yt-dlp/yt-dlp#installation
<!-- Badges -->
[crates.io]: https://crates.io/crates/songbird
[crates.io badge]: https://img.shields.io/crates/v/songbird.svg?style=flat-square
[build]: https://github.com/serenity-rs/songbird/actions
[build badge]: https://img.shields.io/github/actions/workflow/status/serenity-rs/songbird/ci.yml?branch=current&style=flat-square
[docs]: https://docs.rs/songbird/
[docs badge]: https://img.shields.io/badge/docs-online-informational?style=flat-square
[current docs]: https://serenity-rs.github.io/songbird/current/songbird/index.html
[current docs badge]: https://img.shields.io/badge/docs-current-4d76ae.svg?style=flat-square
[next docs]: https://serenity-rs.github.io/songbird/next/songbird/index.html
[next docs badge]: https://img.shields.io/badge/docs-next-4d76ae.svg?style=flat-square
[msrv]: https://blog.rust-lang.org/2024/11/28/Rust-1.83.0.html
[msrv badge]: https://img.shields.io/badge/rust-1.83+-93450a.svg?style=flat-square
[license]: LICENSE
[license badge]: https://img.shields.io/crates/l/songbird.svg?style=flat-square&color=yellow
[guild]: https://discord.gg/serenity-rs
[guild badge]: https://img.shields.io/discord/381880193251409931.svg?style=flat-square&colorB=7289DA

24
app/vendor/songbird/build.rs vendored Normal file
View File

@@ -0,0 +1,24 @@
#[cfg(all(feature = "driver", not(any(feature = "rustls", feature = "native"))))]
compile_error!(
"You have the `driver` feature enabled: \
either the `rustls` or `native` feature must be
selected to let Songbird's driver use websockets.\n\
- `rustls` uses Rustls, a pure Rust TLS-implemenation.\n\
- `native` uses SChannel on Windows, Secure Transport on macOS, \
and OpenSSL on other platforms.\n\
If you are unsure, go with `rustls`."
);
#[cfg(any(
all(feature = "driver", feature = "tws", feature = "tungstenite"),
all(feature = "driver", not(feature = "tws"), not(feature = "tungstenite"))
))]
compile_error!(
"You have the `driver` feature enabled: \
this requires you specify either: \n\
- `tungstenite` (recommended with serenity)\n\
- or `tws` (recommended with twilight).\n\
You have either specified none, or both - choose exactly one."
);
fn main() {}

444
app/vendor/songbird/src/config.rs vendored Normal file
View File

@@ -0,0 +1,444 @@
#[cfg(all(feature = "driver", feature = "receive"))]
use crate::driver::DecodeMode;
use crate::FloatDuration;
#[cfg(feature = "driver")]
use crate::{
driver::{
get_default_scheduler,
retry::Retry,
tasks::disposal::DisposalThread,
CryptoMode,
MixMode,
Scheduler,
},
input::codecs::*,
};
#[cfg(test)]
use crate::driver::test_config::*;
#[cfg(all(test, feature = "driver"))]
use crate::driver::SchedulerConfig;
#[cfg(feature = "driver")]
use symphonia::core::{codecs::CodecRegistry, probe::Probe};
use derivative::Derivative;
#[cfg(all(feature = "driver", feature = "receive"))]
use std::num::NonZeroU8;
use std::time::Duration;
/// Configuration for drivers and calls.
#[derive(Clone, Derivative)]
#[derivative(Debug)]
#[non_exhaustive]
pub struct Config {
#[cfg(feature = "driver")]
/// Selected tagging mode for voice packet encryption.
///
/// Defaults to [`CryptoMode::Aes256Gcm`].
///
/// Changes to this field will not immediately apply if the
/// driver is actively connected, but will apply to subsequent
/// sessions.
///
/// [`CryptoMode::Aes256Gcm`]: CryptoMode::Aes256Gcm
pub crypto_mode: CryptoMode,
#[cfg(all(feature = "driver", feature = "receive"))]
/// Configures whether decoding and decryption occur for all received packets.
///
/// If receiving and using voice packets, generally you should choose [`DecodeMode::Decode`].
/// [`DecodeMode::Decrypt`] is intended for users running their own selective decoding or
/// who need to inspect Opus packets. [User speaking state] can still be seen using [`DecodeMode::Pass`].
/// If you're certain you will never need any RT(C)P events, then consider building without
/// the `"receive"` feature for extra performance.
///
/// Defaults to [`DecodeMode::Decrypt`]. This is due to per-packet decoding costs,
/// which most users will not want to pay, but allowing speaking events which are commonly used.
///
/// [`DecodeMode::Decode`]: DecodeMode::Decode
/// [`DecodeMode::Decrypt`]: DecodeMode::Decrypt
/// [`DecodeMode::Pass`]: DecodeMode::Pass
/// [User speaking state]: crate::events::CoreEvent::VoiceTick
pub decode_mode: DecodeMode,
#[cfg(all(feature = "driver", feature = "receive"))]
/// Configures the amount of time after a user/SSRC is inactive before their decoder state
/// should be removed.
///
/// Defaults to 1 minute.
pub decode_state_timeout: FloatDuration,
#[cfg(all(feature = "driver", feature = "receive"))]
/// Configures the number of audio packets to buffer for each user before playout.
///
/// A playout buffer allows Songbird to smooth out jitter in audio packet arrivals,
/// as well as to correct for reordering of packets by the network.
///
/// This does not affect the arrival of raw packet events.
///
/// Defaults to 5 packets (100ms).
pub playout_buffer_length: NonZeroU8,
#[cfg(all(feature = "driver", feature = "receive"))]
/// Configures the initial amount of extra space allocated to handle packet bursts.
///
/// Each SSRC's receive buffer will start at capacity `playout_buffer_length +
/// playout_spike_length`, up to a maximum 64 packets.
///
/// Defaults to 3 packets (thus capacity defaults to 8).
pub playout_spike_length: u8,
#[cfg(feature = "gateway")]
/// Configures the amount of time to wait for Discord to reply with connection information
/// if [`Call::join`]/[`join_gateway`] are used.
///
/// This is a useful fallback in the event that:
/// * the underlying Discord client restarts and loses a join request, or
/// * a channel join fails because the bot is already believed to be there.
///
/// Defaults to 10 seconds. If set to `None`, connections will never time out.
///
/// [`Call::join`]: crate::Call::join
/// [`join_gateway`]: crate::Call::join_gateway
pub gateway_timeout: Option<FloatDuration>,
#[cfg(feature = "driver")]
/// Configures whether the driver will mix and output stereo or mono Opus data
/// over a voice channel.
///
/// Defaults to [`Stereo`].
///
/// [`Stereo`]: MixMode::Stereo
pub mix_mode: MixMode,
#[cfg(feature = "driver")]
/// Number of concurrently active tracks to allocate memory for.
///
/// This should be set at, or just above, the maximum number of tracks
/// you expect your bot will play at the same time. Exceeding the size of
/// the internal queue will trigger a larger memory allocation and copy,
/// possibly causing the mixer thread to miss a packet deadline.
///
/// Defaults to `1`.
///
/// Changes to this field in a running driver will only ever increase
/// the capacity of the track store.
pub preallocated_tracks: u8,
#[cfg(feature = "driver")]
/// Connection retry logic for the [`Driver`].
///
/// This controls how many times the [`Driver`] should retry any connections,
/// as well as how long to wait between attempts.
///
/// [`Driver`]: crate::driver::Driver
pub driver_retry: Retry,
#[cfg(feature = "driver")]
/// Configures whether or not each mixed audio packet is [soft-clipped] into the
/// [-1, 1] audio range.
///
/// Defaults to `true`, preventing clipping and dangerously loud audio from being sent.
///
/// **This operation adds ~3% cost to a standard (non-passthrough) mix cycle.**
/// If you *know* that your bot will only play one sound at a time and that
/// your volume is between `0.0` and `1.0`, then you can disable soft-clipping
/// for a performance boost. If you are playing several sounds at once, do not
/// disable this unless you make sure to reduce the volume of each sound.
///
/// [soft-clipped]: https://opus-codec.org/docs/opus_api-1.3.1/group__opus__decoder.html#gaff99598b352e8939dded08d96e125e0b
pub use_softclip: bool,
#[cfg(feature = "driver")]
/// Configures the maximum amount of time to wait for an attempted voice
/// connection to Discord.
///
/// Defaults to 10 seconds. If set to `None`, connections will never time out.
pub driver_timeout: Option<FloatDuration>,
#[cfg(feature = "driver")]
#[derivative(Debug = "ignore")]
/// Registry of the inner codecs supported by the driver, adding opus2-based
/// Opus codec support to all of Symphonia's default codecs.
///
/// Defaults to [`get_codec_registry`].
pub codec_registry: &'static CodecRegistry,
#[cfg(feature = "driver")]
#[derivative(Debug = "ignore")]
/// Registry of the muxers and container formats supported by the driver.
///
/// Defaults to [`get_probe`], which includes all of Symphonia's default format handlers
/// and DCA format support.
pub format_registry: &'static Probe,
#[cfg(feature = "driver")]
/// The Sender for a channel that will run the destructor of possibly blocking values.
///
/// If not set, a thread will be spawned to perform this, but it is recommended to create
/// a long running thread instead of relying on a per-driver thread.
///
/// Note: When using [`Songbird`] this is overwritten automatically by its disposal thread.
///
/// [`Songbird`]: crate::Songbird
pub disposer: Option<DisposalThread>,
#[cfg(feature = "driver")]
/// The scheduler is responsible for mapping idle and active [`Driver`] instances
/// to threads.
///
/// If set to None, then songbird will use [`get_default_scheduler`].
///
/// [`Driver`]: crate::Driver
pub scheduler: Option<Scheduler>,
// Test only attributes
#[cfg(feature = "driver")]
#[cfg(test)]
/// Test config to offer precise control over mixing tick rate/count.
pub(crate) tick_style: TickStyle,
#[cfg(feature = "driver")]
#[cfg(test)]
/// If set, skip connection and encryption steps.
pub(crate) override_connection: Option<OutputMode>,
}
impl Default for Config {
fn default() -> Self {
Self {
#[cfg(feature = "driver")]
crypto_mode: CryptoMode::Aes256Gcm,
#[cfg(all(feature = "driver", feature = "receive"))]
decode_mode: DecodeMode::Decrypt,
#[cfg(all(feature = "driver", feature = "receive"))]
decode_state_timeout: Duration::from_secs(60).into(),
#[cfg(all(feature = "driver", feature = "receive"))]
playout_buffer_length: NonZeroU8::new(5).unwrap(),
#[cfg(all(feature = "driver", feature = "receive"))]
playout_spike_length: 3,
#[cfg(feature = "gateway")]
gateway_timeout: Some(Duration::from_secs(10).into()),
#[cfg(feature = "driver")]
mix_mode: MixMode::Stereo,
#[cfg(feature = "driver")]
preallocated_tracks: 1,
#[cfg(feature = "driver")]
use_softclip: true,
#[cfg(feature = "driver")]
driver_retry: Retry::default(),
#[cfg(feature = "driver")]
driver_timeout: Some(Duration::from_secs(10).into()),
#[cfg(feature = "driver")]
codec_registry: get_codec_registry(),
#[cfg(feature = "driver")]
format_registry: get_probe(),
#[cfg(feature = "driver")]
disposer: None,
#[cfg(feature = "driver")]
scheduler: None,
#[cfg(feature = "driver")]
#[cfg(test)]
tick_style: TickStyle::Timed,
#[cfg(feature = "driver")]
#[cfg(test)]
override_connection: None,
}
}
}
#[cfg(feature = "driver")]
impl Config {
/// Sets this `Config`'s chosen cryptographic tagging scheme.
#[must_use]
pub fn crypto_mode(mut self, crypto_mode: CryptoMode) -> Self {
self.crypto_mode = crypto_mode;
self
}
#[cfg(feature = "receive")]
/// Sets this `Config`'s received packet decryption/decoding behaviour.
#[must_use]
pub fn decode_mode(mut self, decode_mode: DecodeMode) -> Self {
self.decode_mode = decode_mode;
self
}
#[cfg(feature = "receive")]
/// Sets this `Config`'s received packet decoder cleanup timer.
#[must_use]
pub fn decode_state_timeout(mut self, decode_state_timeout: Duration) -> Self {
self.decode_state_timeout = decode_state_timeout.into();
self
}
#[cfg(feature = "receive")]
/// Sets this `Config`'s playout buffer length, in packets.
#[must_use]
pub fn playout_buffer_length(mut self, playout_buffer_length: NonZeroU8) -> Self {
self.playout_buffer_length = playout_buffer_length;
self
}
#[cfg(feature = "receive")]
/// Sets this `Config`'s additional pre-allocated space to handle bursty audio packets.
#[must_use]
pub fn playout_spike_length(mut self, playout_spike_length: u8) -> Self {
self.playout_spike_length = playout_spike_length;
self
}
/// Sets this `Config`'s audio mixing channel count.
#[must_use]
pub fn mix_mode(mut self, mix_mode: MixMode) -> Self {
self.mix_mode = mix_mode;
self
}
/// Sets this `Config`'s number of tracks to preallocate.
#[must_use]
pub fn preallocated_tracks(mut self, preallocated_tracks: u8) -> Self {
self.preallocated_tracks = preallocated_tracks;
self
}
/// Sets this `Config`'s number to enable/disable soft-clipping sent audio.
#[must_use]
pub fn use_softclip(mut self, use_softclip: bool) -> Self {
self.use_softclip = use_softclip;
self
}
/// Sets this `Config`'s timeout for establishing a voice connection.
#[must_use]
pub fn driver_timeout(mut self, driver_timeout: Option<Duration>) -> Self {
self.driver_timeout = driver_timeout.map(Into::into);
self
}
/// Sets this `Config`'s voice connection retry configuration.
#[must_use]
pub fn driver_retry(mut self, driver_retry: Retry) -> Self {
self.driver_retry = driver_retry;
self
}
/// Sets this `Config`'s symphonia codec registry.
#[must_use]
pub fn codec_registry(mut self, codec_registry: &'static CodecRegistry) -> Self {
self.codec_registry = codec_registry;
self
}
/// Sets this `Config`'s symphonia format registry/probe set.
#[must_use]
pub fn format_registry(mut self, format_registry: &'static Probe) -> Self {
self.format_registry = format_registry;
self
}
/// Sets this `Config`'s channel for sending disposal messages.
#[must_use]
pub fn disposer(mut self, disposer: DisposalThread) -> Self {
self.disposer = Some(disposer);
self
}
/// Sets this `Config`'s mixer scheduler.
#[must_use]
pub fn scheduler(mut self, scheduler: Scheduler) -> Self {
self.scheduler = Some(scheduler);
self
}
/// Returns a lightweight reference to the audio scheduler this `Config` will use.
#[must_use]
pub fn get_scheduler(&self) -> Scheduler {
self.scheduler
.as_ref()
.unwrap_or(get_default_scheduler())
.clone()
}
/// Ensures a global disposer has been set, initializing one if not.
#[must_use]
pub(crate) fn initialise_disposer(self) -> Self {
if self.disposer.is_some() {
self
} else {
self.disposer(DisposalThread::run())
}
}
/// This is used to prevent changes which would invalidate the current session.
pub(crate) fn make_safe(&mut self, previous: &Config, connected: bool) {
if connected {
self.crypto_mode = previous.crypto_mode;
}
}
}
#[cfg(not(feature = "driver"))]
impl Config {
pub(crate) fn initialise_disposer(self) -> Self {
self
}
}
// Test only attributes
#[cfg(all(test, feature = "driver"))]
#[allow(missing_docs)]
impl Config {
#[must_use]
pub fn tick_style(mut self, tick_style: TickStyle) -> Self {
self.tick_style = tick_style;
self
}
/// Sets this `Config`'s voice connection retry configuration.
#[must_use]
pub fn override_connection(mut self, override_connection: Option<OutputMode>) -> Self {
self.override_connection = override_connection;
self
}
#[must_use]
pub fn test_cfg(raw_output: bool) -> (DriverTestHandle, Config) {
let (tick_tx, tick_rx) = flume::unbounded();
let (conn, rx) = if raw_output {
let (pkt_tx, pkt_rx) = flume::unbounded();
(OutputMode::Raw(pkt_tx), OutputReceiver::Raw(pkt_rx))
} else {
let (rtp_tx, rtp_rx) = flume::unbounded();
(OutputMode::Rtp(rtp_tx), OutputReceiver::Rtp(rtp_rx))
};
let sc_config = SchedulerConfig {
strategy: crate::driver::SchedulerMode::MaxPerThread(1.try_into().unwrap()),
move_expensive_tasks: true,
};
let config = Config::default()
.tick_style(TickStyle::UntimedWithExecLimit(tick_rx))
// give each test its own thread in the scheduler for simplicity.
.scheduler(Scheduler::new(sc_config))
.override_connection(Some(conn));
let handle = DriverTestHandle { rx, tx: tick_tx };
(handle, config)
}
}
#[cfg(feature = "gateway")]
impl Config {
/// Sets this `Config`'s timeout for joining a voice channel.
#[must_use]
pub fn gateway_timeout(mut self, gateway_timeout: Option<Duration>) -> Self {
self.gateway_timeout = gateway_timeout.map(Into::into);
self
}
}

136
app/vendor/songbird/src/constants.rs vendored Normal file
View File

@@ -0,0 +1,136 @@
//! Constants affecting driver function and API handling.
#[cfg(feature = "driver")]
use discortp::rtp::RtpType;
#[cfg(feature = "driver")]
use opus2::Bitrate;
use std::time::Duration;
#[cfg(feature = "driver")]
/// The voice gateway version used by the library.
pub const VOICE_GATEWAY_VERSION: u8 = crate::model::constants::GATEWAY_VERSION;
#[cfg(feature = "driver")]
/// Sample rate of audio to be sent to Discord.
pub const SAMPLE_RATE: u32 = 48000;
/// Sample rate of audio to be sent to Discord.
pub const SAMPLE_RATE_RAW: usize = 48_000;
/// Number of audio frames/packets to be sent per second.
pub const AUDIO_FRAME_RATE: usize = 50;
/// Length of time between any two audio frames.
pub const TIMESTEP_LENGTH: Duration = Duration::from_millis(1000 / AUDIO_FRAME_RATE as u64);
#[cfg(feature = "driver")]
/// Default bitrate for audio.
pub const DEFAULT_BITRATE: Bitrate = Bitrate::Bits(128_000);
/// Number of output samples at 48kHZ to produced when resampling subframes.
pub(crate) const RESAMPLE_OUTPUT_FRAME_SIZE: usize = MONO_FRAME_SIZE / 2;
/// The maximum number of bad frames to allow in an Opus source before blocking passthrough.
pub(crate) const OPUS_PASSTHROUGH_STRIKE_LIMIT: u8 = 3;
/// Number of samples in one complete frame of audio per channel.
///
/// This is equally the number of stereo (joint) samples in an audio frame.
pub const MONO_FRAME_SIZE: usize = SAMPLE_RATE_RAW / AUDIO_FRAME_RATE;
/// Number of individual samples in one complete frame of stereo audio.
pub const STEREO_FRAME_SIZE: usize = 2 * MONO_FRAME_SIZE;
/// Number of bytes in one complete frame of raw `f32`-encoded mono audio.
pub const MONO_FRAME_BYTE_SIZE: usize = MONO_FRAME_SIZE * std::mem::size_of::<f32>();
/// Number of bytes in one complete frame of raw `f32`-encoded stereo audio.
pub const STEREO_FRAME_BYTE_SIZE: usize = STEREO_FRAME_SIZE * std::mem::size_of::<f32>();
/// Length (in milliseconds) of any audio frame.
pub const FRAME_LEN_MS: usize = 1000 / AUDIO_FRAME_RATE;
/// Maximum number of audio frames/packets to be sent per second to be buffered.
pub const CHILD_BUFFER_LEN: usize = AUDIO_FRAME_RATE / 2;
/// Maximum packet size for a voice packet.
///
/// Set a safe amount below the Ethernet MTU to avoid fragmentation/rejection.
pub const VOICE_PACKET_MAX: usize = 1460;
/// Delay between sends of UDP keepalive frames.
///
/// Passive monitoring of Discord itself shows that these fire every 5 seconds
/// irrespective of outgoing UDP traffic.
pub const UDP_KEEPALIVE_GAP_MS: u64 = 5_000;
/// Type-converted delay between sends of UDP keepalive frames.
///
/// Passive monitoring of Discord itself shows that these fire every 5 seconds
/// irrespective of outgoing UDP traffic.
pub const UDP_KEEPALIVE_GAP: Duration = Duration::from_millis(UDP_KEEPALIVE_GAP_MS);
/// Opus silent frame, used to signal speech start and end (and prevent audio glitching).
pub const SILENT_FRAME: [u8; 3] = [0xf8, 0xff, 0xfe];
/// The one (and only) RTP version.
pub const RTP_VERSION: u8 = 2;
#[cfg(feature = "driver")]
/// Profile type used by Discord's Opus audio traffic.
pub const RTP_PROFILE_TYPE: RtpType = RtpType::Dynamic(120);
#[cfg(feature = "receive")]
/// The magic marker appended to the end of DAVE encrypted payloads, used to determine
/// whether the voice packet is encrypted.
pub const DAVE_MAGIC_MARKER: [u8; 2] = [0xfa, 0xfa];
#[cfg(test)]
#[allow(clippy::doc_markdown, missing_docs)]
pub mod test_data {
/// URL for a source which YTDL must extract.
///
/// Referenced under CC BY-NC-SA 3.0 -- https://creativecommons.org/licenses/by-nc-sa/3.0/
pub const YTDL_TARGET: &str = "https://cloudkicker.bandcamp.com/track/94-days";
/// URL for a source that has both a playlist and a music video,
/// which YTDL should extract.
///
/// Referenced under CC BY-NC-SA 3.0 -- https://creativecommons.org/licenses/by-nc-sa/3.0/
pub const YTDL_PLAYLIST_TARGET: &str =
"https://www.youtube.com/watch?v=KSgEFfWZ-W0&list=OLAK5uy_l2x81ffbpevMSjUn7NniL_rNLulWM3n6g&index=7";
/// URL for a source which can be read via an Http Request.
///
/// Referenced under CC BY-NC-SA 3.0 -- https://creativecommons.org/licenses/by-nc-sa/3.0/
pub const HTTP_TARGET: &str = "https://github.com/FelixMcFelix/songbird/raw/symphonia/resources/Cloudkicker%20-%202011%2007.mp3";
/// URL for an opus/ogg source which can be read via an Http Request.
///
/// Referenced under CC BY 3.0 -- https://creativecommons.org/licenses/by/3.0/
pub const HTTP_OPUS_TARGET: &str = "https://github.com/FelixMcFelix/songbird/raw/symphonia/resources/Cloudkicker%20-%20Making%20Will%20Mad.opus";
/// URL for an opus/webm source which can be read via an Http Request.
///
/// Referenced under CC BY 3.0 -- https://creativecommons.org/licenses/by/3.0/
pub const HTTP_WEBM_TARGET: &str = "https://github.com/FelixMcFelix/songbird/raw/symphonia/resources/Cloudkicker%20-%20Making%20Will%20Mad.webm";
/// Path to a DCA source.
///
/// Referenced under CC BY-NC-SA 3.0 -- https://creativecommons.org/licenses/by-nc-sa/3.0/
pub const FILE_DCA_TARGET: &str = "resources/Cloudkicker - 2011 07.dca1";
/// Path to an opus source which can be read via a File.
///
/// Referenced under CC BY 3.0 -- https://creativecommons.org/licenses/by/3.0/
pub const FILE_WEBM_TARGET: &str = "resources/Cloudkicker - Making Will Mad.webm";
/// Path to a Wav source which can be read via a File.
pub const FILE_WAV_TARGET: &str = "resources/loop.wav";
/// Path to a shorter MP3 source which can be read via a File.
pub const FILE_SHORT_MP3_TARGET: &str = "resources/ting.mp3";
/// Path to an MP4 (H264 + AAC) source which can be read via a File.
pub const FILE_VID_TARGET: &str = "resources/ting-vid.mp4";
}

View File

@@ -0,0 +1,24 @@
//! Various driver internals which need to be exported for benchmarking.
//!
//! Included if using the `"internals"` feature flag.
//! You should not and/or cannot use these as part of a normal application.
#![allow(missing_docs)]
pub use super::tasks::{message as task_message, mixer};
pub use super::crypto::{Cipher, CryptoState};
use crate::{
driver::tasks::message::TrackContext,
tracks::{Track, TrackHandle},
};
#[must_use]
pub fn track_context(t: Track) -> (TrackHandle, TrackContext) {
t.into_context()
}
pub mod scheduler {
pub use crate::driver::scheduler::*;
}

View File

@@ -0,0 +1,152 @@
//! Connection errors and convenience types.
use crate::{
driver::tasks::{error::Recipient, message::*},
ws::Error as WsError,
};
use aes_gcm::Error as CryptoError;
use davey::errors::{
CreateKeyPackageError as DaveyCreateKeyPackageError,
InitError as DaveyInitError,
};
use flume::SendError;
use serde_json::Error as JsonError;
use std::{error::Error as StdError, fmt, io::Error as IoError};
use tokio::time::error::Elapsed;
/// Errors encountered while connecting to a Discord voice server over the driver.
#[derive(Debug)]
#[non_exhaustive]
pub enum Error {
/// The driver hung up an internal signaller, either due to another connection attempt
/// or a crash.
AttemptDiscarded,
/// An error occurred during [en/de]cryption of voice packets.
Crypto(CryptoError),
/// The symmetric key supplied by Discord had the wrong size.
CryptoInvalidLength,
/// Server did not return the expected crypto mode during negotiation.
CryptoModeInvalid,
/// Selected crypto mode was not offered by server.
CryptoModeUnavailable,
/// Failed to create a DAVE key package.
#[cfg(feature = "driver")]
DaveCreateKeyPackageError(DaveyCreateKeyPackageError),
/// An error occured during initialization of the DAVE session.
#[cfg(feature = "driver")]
DaveInitializationError(DaveyInitError),
/// An indicator that an endpoint URL was invalid.
EndpointUrl,
/// Discord failed to correctly respond to IP discovery.
IllegalDiscoveryResponse,
/// Could not parse Discord's view of our IP.
IllegalIp,
/// Miscellaneous I/O error.
Io(IoError),
/// JSON (de)serialization error.
Json(JsonError),
/// Failed to message other background tasks after connection establishment.
InterconnectFailure(Recipient),
/// Error communicating with gateway server over WebSocket.
Ws(WsError),
/// Connection attempt timed out.
TimedOut,
}
impl From<CryptoError> for Error {
fn from(e: CryptoError) -> Self {
Error::Crypto(e)
}
}
impl From<IoError> for Error {
fn from(e: IoError) -> Error {
Error::Io(e)
}
}
impl From<JsonError> for Error {
fn from(e: JsonError) -> Error {
Error::Json(e)
}
}
impl From<SendError<WsMessage>> for Error {
fn from(_e: SendError<WsMessage>) -> Error {
Error::InterconnectFailure(Recipient::AuxNetwork)
}
}
impl From<SendError<EventMessage>> for Error {
fn from(_e: SendError<EventMessage>) -> Error {
Error::InterconnectFailure(Recipient::Event)
}
}
impl From<SendError<MixerMessage>> for Error {
fn from(_e: SendError<MixerMessage>) -> Error {
Error::InterconnectFailure(Recipient::Mixer)
}
}
impl From<WsError> for Error {
fn from(e: WsError) -> Error {
Error::Ws(e)
}
}
impl From<Elapsed> for Error {
fn from(_e: Elapsed) -> Error {
Error::TimedOut
}
}
impl fmt::Display for Error {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "failed to connect to Discord RTP server: ")?;
match self {
Self::AttemptDiscarded => write!(f, "connection attempt was aborted/discarded"),
Self::Crypto(e) => e.fmt(f),
Self::CryptoInvalidLength => write!(f, "server supplied key of wrong length"),
Self::CryptoModeInvalid => write!(f, "server changed negotiated encryption mode"),
Self::CryptoModeUnavailable => write!(f, "server did not offer chosen encryption mode"),
Self::DaveCreateKeyPackageError(e) => e.fmt(f),
Self::DaveInitializationError(e) => e.fmt(f),
Self::EndpointUrl => write!(f, "endpoint URL received from gateway was invalid"),
Self::IllegalDiscoveryResponse => {
write!(f, "IP discovery/NAT punching response was invalid")
},
Self::IllegalIp => write!(f, "IP discovery/NAT punching response had bad IP value"),
Self::Io(e) => e.fmt(f),
Self::Json(e) => e.fmt(f),
Self::InterconnectFailure(e) => write!(f, "failed to contact other task ({e:?})"),
Self::Ws(e) => write!(f, "websocket issue ({e:?})."),
Self::TimedOut => write!(f, "connection attempt timed out"),
}
}
}
impl StdError for Error {
fn source(&self) -> Option<&(dyn StdError + 'static)> {
match self {
Error::AttemptDiscarded
| Error::Crypto(_)
| Error::CryptoInvalidLength
| Error::CryptoModeInvalid
| Error::CryptoModeUnavailable
| Error::DaveCreateKeyPackageError(_)
| Error::DaveInitializationError(_)
| Error::EndpointUrl
| Error::IllegalDiscoveryResponse
| Error::IllegalIp
| Error::InterconnectFailure(_)
| Error::Ws(_)
| Error::TimedOut => None,
Error::Io(e) => e.source(),
Error::Json(e) => e.source(),
}
}
}
/// Convenience type for Discord voice/driver connection error handling.
pub type Result<T> = std::result::Result<T, Error>;

View File

@@ -0,0 +1,413 @@
pub mod error;
#[cfg(feature = "receive")]
use super::tasks::udp_rx;
use super::{
crypto::Cipher,
tasks::{
message::*,
ws::{self as ws_task, AuxNetwork},
},
Config,
CryptoMode,
};
use crate::{
constants::*,
model::{
payload::{Identify, Resume, SelectProtocol},
Event as GatewayEvent,
ProtocolData,
},
ws::WsStream,
ConnectionInfo,
};
use discortp::discord::{IpDiscoveryPacket, IpDiscoveryType, MutableIpDiscoveryPacket};
use error::{Error, Result};
use flume::Sender;
use serenity_voice_model::payload::DaveMlsKeyPackage;
use socket2::Socket;
use std::sync::{atomic::AtomicU16, Arc, RwLock};
use std::{net::IpAddr, num::NonZeroU16, str::FromStr};
use tokio::{net::UdpSocket, spawn, time::timeout};
use tracing::{debug, info, instrument};
use url::Url;
pub(crate) struct Connection {
pub(crate) info: ConnectionInfo,
pub(crate) ssrc: u32,
pub(crate) ws: Sender<WsMessage>,
}
impl Connection {
pub(crate) async fn new(
info: ConnectionInfo,
interconnect: &Interconnect,
config: &Config,
idx: usize,
) -> Result<Connection> {
if let Some(t) = config.driver_timeout {
timeout(
t.into(),
Connection::new_inner(info, interconnect, config, idx),
)
.await?
} else {
Connection::new_inner(info, interconnect, config, idx).await
}
}
pub(crate) async fn new_inner(
info: ConnectionInfo,
interconnect: &Interconnect,
config: &Config,
idx: usize,
) -> Result<Connection> {
let url = generate_url(&info.endpoint)?;
let mut client = WsStream::connect(url).await?;
let (ws_msg_tx, ws_msg_rx) = flume::unbounded();
let mut hello = None;
let mut ready = None;
client
.send_json(&GatewayEvent::from(Identify {
server_id: info.guild_id.into(),
session_id: info.session_id.clone(),
token: info.token.clone(),
user_id: info.user_id.into(),
max_dave_protocol_version: Some(davey::DAVE_PROTOCOL_VERSION),
}))
.await?;
loop {
let Some(value) = client.recv_event().await? else {
continue;
};
match value {
GatewayEvent::Ready(r) => {
ready = Some(r);
if hello.is_some() {
break;
}
},
GatewayEvent::Hello(h) => {
hello = Some(h);
if ready.is_some() {
break;
}
},
other => {
// Discord hold back per-user connection state until after this handshake.
// There's no guarantee that will remain the case, so buffer it like all
// subsequent steps where we know they *do* send these packets.
debug!("Expected ready/hello; got: {:?}", other);
ws_msg_tx.send(WsMessage::Deliver(other))?;
},
}
}
let hello =
hello.expect("Hello packet expected in connection initialisation, but not found.");
let ready =
ready.expect("Ready packet expected in connection initialisation, but not found.");
let chosen_crypto = CryptoMode::negotiate(&ready.modes, Some(config.crypto_mode))?;
info!(
"Crypto scheme negotiation -- wanted {:?}. Chose {:?} from modes {:?}.",
config.crypto_mode, chosen_crypto, ready.modes
);
let udp = UdpSocket::bind("0.0.0.0:0").await?;
// Optimisation for non-receive case: set rx buffer size to zero.
let udp = if cfg!(feature = "receive") {
udp
} else {
let socket = Socket::from(udp.into_std()?);
// Some operating systems do not allow setting the recv buffer to 0.
#[cfg(any(target_os = "linux", target_os = "windows"))]
socket.set_recv_buffer_size(0)?;
UdpSocket::from_std(socket.into())?
};
udp.connect((ready.ip, ready.port)).await?;
// Follow Discord's IP Discovery procedures, in case NAT tunnelling is needed.
let mut bytes = [0; IpDiscoveryPacket::const_packet_size()];
{
let mut view = MutableIpDiscoveryPacket::new(&mut bytes[..]).expect(
"Too few bytes in 'bytes' for IPDiscovery packet.\
(Blame: IpDiscoveryPacket::const_packet_size()?)",
);
view.set_pkt_type(IpDiscoveryType::Request);
view.set_length(70);
view.set_ssrc(ready.ssrc);
}
udp.send(&bytes).await?;
let (len, _addr) = udp.recv_from(&mut bytes).await?;
{
let view =
IpDiscoveryPacket::new(&bytes[..len]).ok_or(Error::IllegalDiscoveryResponse)?;
if view.get_pkt_type() != IpDiscoveryType::Response {
return Err(Error::IllegalDiscoveryResponse);
}
// We could do something clever like binary search,
// but possibility of UDP spoofing precludes us from
// making the assumption we can find a "left edge" of '\0's.
let nul_byte_index = view
.get_address_raw()
.iter()
.position(|&b| b == 0)
.ok_or(Error::IllegalIp)?;
let address_str = std::str::from_utf8(&view.get_address_raw()[..nul_byte_index])
.map_err(|_| Error::IllegalIp)?;
let address = IpAddr::from_str(address_str).map_err(|_| Error::IllegalIp)?;
client
.send_json(&GatewayEvent::from(SelectProtocol {
protocol: "udp".into(),
data: ProtocolData {
address,
mode: chosen_crypto.to_request_str().into(),
port: view.get_port(),
},
}))
.await?;
}
let (cipher, dave_session, dave_protocol_version) =
init_cipher(&mut client, &info, chosen_crypto, &ws_msg_tx).await?;
let dave_session = Arc::new(RwLock::new(dave_session));
let dave_protocol_version = Arc::new(dave_protocol_version);
info!("Connected to: {}", info.endpoint);
info!("WS heartbeat duration {}ms.", hello.heartbeat_interval);
#[cfg(feature = "receive")]
let (udp_receiver_msg_tx, udp_receiver_msg_rx) = flume::unbounded();
// NOTE: This causes the UDP Socket on "receive" to be non-blocking,
// and the standard to be blocking. A UDP send should only WouldBlock if
// you're sending more data than the OS can handle (not likely, and
// at that point you should scale horizontally).
//
// If this is a problem for anyone, we can make non-blocking sends
// queue up a delayed send up to a limit.
#[cfg(feature = "receive")]
let (udp_rx, udp_tx) = {
let udp_tx = udp.into_std()?;
let udp_rx = UdpSocket::from_std(udp_tx.try_clone()?)?;
(udp_rx, udp_tx)
};
#[cfg(not(feature = "receive"))]
let udp_tx = udp.into_std()?;
let ssrc = ready.ssrc;
let mix_conn = MixerConnection {
#[cfg(feature = "receive")]
cipher: cipher.clone(),
#[cfg(not(feature = "receive"))]
cipher,
dave_session: dave_session.clone(),
dave_protocol_version: dave_protocol_version.clone(),
crypto_state: chosen_crypto.into(),
#[cfg(feature = "receive")]
udp_rx: udp_receiver_msg_tx,
udp_tx,
};
interconnect
.mixer
.send(MixerMessage::Ws(Some(ws_msg_tx.clone())))?;
interconnect
.mixer
.send(MixerMessage::SetConn(mix_conn, ready.ssrc))?;
#[cfg(feature = "receive")]
let ssrc_tracker = Arc::new(SsrcTracker::default());
let ws_state = AuxNetwork::new(
ws_msg_rx,
client,
ssrc,
hello.heartbeat_interval,
idx,
info.clone(),
#[cfg(not(feature = "receive"))]
dave_session,
#[cfg(not(feature = "receive"))]
dave_protocol_version,
#[cfg(feature = "receive")]
dave_session.clone(),
#[cfg(feature = "receive")]
dave_protocol_version.clone(),
#[cfg(feature = "receive")]
ssrc_tracker.clone(),
);
spawn(ws_task::runner(interconnect.clone(), ws_state));
#[cfg(feature = "receive")]
spawn(udp_rx::runner(
interconnect.clone(),
udp_receiver_msg_rx,
cipher,
chosen_crypto,
config.clone(),
udp_rx,
ssrc_tracker,
dave_session,
dave_protocol_version,
));
Ok(Connection {
info,
ssrc,
ws: ws_msg_tx,
})
}
#[instrument(skip(self))]
pub async fn reconnect(&mut self, config: &Config) -> Result<()> {
if let Some(t) = config.driver_timeout {
timeout(t.into(), self.reconnect_inner()).await?
} else {
self.reconnect_inner().await
}
}
#[instrument(skip(self))]
pub async fn reconnect_inner(&mut self) -> Result<()> {
let url = generate_url(&self.info.endpoint)?;
// Thread may have died, we want to send to prompt a clean exit
// (if at all possible) and then proceed as normal.
let mut client = WsStream::connect(url).await?;
client
.send_json(&GatewayEvent::from(Resume {
server_id: self.info.guild_id.into(),
session_id: self.info.session_id.clone(),
token: self.info.token.clone(),
}))
.await?;
let mut hello = None;
let mut resumed = None;
loop {
let Some(value) = client.recv_event().await? else {
continue;
};
match value {
GatewayEvent::Resumed => {
resumed = Some(());
if hello.is_some() {
break;
}
},
GatewayEvent::Hello(h) => {
hello = Some(h);
if resumed.is_some() {
break;
}
},
other => {
self.ws.send(WsMessage::Deliver(other))?;
},
}
}
let hello =
hello.expect("Hello packet expected in connection initialisation, but not found.");
self.ws
.send(WsMessage::SetKeepalive(hello.heartbeat_interval))?;
self.ws.send(WsMessage::Ws(Box::new(client)))?;
info!("Reconnected to: {}", &self.info.endpoint);
Ok(())
}
}
impl Drop for Connection {
fn drop(&mut self) {
info!("Disconnected");
}
}
fn generate_url(endpoint: &str) -> Result<Url> {
Url::parse(&format!("wss://{endpoint}/?v={VOICE_GATEWAY_VERSION}")).or(Err(Error::EndpointUrl))
}
#[inline]
async fn init_cipher(
client: &mut WsStream,
info: &ConnectionInfo,
mode: CryptoMode,
tx: &Sender<WsMessage>,
) -> Result<(Cipher, Option<davey::DaveSession>, AtomicU16)> {
loop {
let Some(value) = client.recv_event().await? else {
continue;
};
match value {
GatewayEvent::SessionDescription(desc) => {
if desc.mode != mode.to_request_str() {
return Err(Error::CryptoModeInvalid);
}
let dave_session =
if let Some(version) = NonZeroU16::new(desc.dave_protocol_version) {
let mut session = davey::DaveSession::new(
version,
info.user_id.0.into(),
info.channel_id.0.into(),
None,
)
.map_err(Error::DaveInitializationError)?;
client
.send_binary(&GatewayEvent::DaveMlsKeyPackage(DaveMlsKeyPackage {
key_package: session
.create_key_package()
.map_err(Error::DaveCreateKeyPackageError)?,
}))
.await?;
Some(session)
} else {
None
};
return Ok((
mode.cipher_from_key(&desc.secret_key)
.map_err(|_| Error::CryptoInvalidLength)?,
dave_session,
AtomicU16::new(desc.dave_protocol_version),
));
},
other => {
// Discord can and will send user-specific payload packets during this time
// which are needed to map SSRCs to `UserId`s.
tx.send(WsMessage::Deliver(other))?;
},
}
}
}

578
app/vendor/songbird/src/driver/crypto.rs vendored Normal file
View File

@@ -0,0 +1,578 @@
//! Encryption schemes supported by Discord's secure RTP negotiation.
#[cfg(any(feature = "receive", test))]
use super::tasks::error::Error as InternalError;
use aead::AeadCore;
use aes_gcm::{AeadInPlace, Aes256Gcm, Error as CryptoError};
use byteorder::{NetworkEndian, WriteBytesExt};
use chacha20poly1305::XChaCha20Poly1305;
use crypto_common::{InvalidLength, KeyInit};
#[cfg(feature = "receive")]
use discortp::rtcp::MutableRtcpPacket;
use discortp::MutablePacket;
#[cfg(any(feature = "receive", test))]
use discortp::{
rtp::{MutableRtpPacket, RtpExtensionPacket},
Packet,
};
use std::{num::Wrapping, str::FromStr};
use typenum::Unsigned;
use crate::error::ConnectionError;
/// Encryption schemes supportd by Discord.
#[derive(Clone, Copy, Debug, Eq, PartialEq, Default, Hash)]
#[non_exhaustive]
pub enum CryptoMode {
#[default]
/// Discord's currently preferred non-E2EE encryption scheme.
///
/// Packets are encrypted and decrypted using the `AES256GCM` encryption scheme.
/// An additional random 4B suffix is used as the source of nonce bytes for the packet.
/// This nonce value increments by `1` with each packet.
///
/// Encrypted content begins *after* the RTP header and extensions, following the SRTP
/// specification.
///
/// Nonce width of 4B (32b), at an extra 4B per packet (~0.2 kB/s).
Aes256Gcm,
/// A fallback non-E2EE encryption scheme.
///
/// Packets are encrypted and decrypted using the `XChaCha20Poly1305` encryption scheme.
/// An additional random 4B suffix is used as the source of nonce bytes for the packet.
/// This nonce value increments by `1` with each packet.
///
/// Encrypted content begins *after* the RTP header and extensions, following the SRTP
/// specification.
///
/// Nonce width of 4B (32b), at an extra 4B per packet (~0.2 kB/s).
XChaCha20Poly1305,
}
impl From<CryptoState> for CryptoMode {
fn from(val: CryptoState) -> Self {
match val {
CryptoState::Aes256Gcm(_) => Self::Aes256Gcm,
CryptoState::XChaCha20Poly1305(_) => Self::XChaCha20Poly1305,
}
}
}
/// The input string could not be parsed as an encryption scheme supported by songbird.
#[derive(Clone, Copy, Debug, Eq, PartialEq, Hash)]
pub struct UnrecognisedCryptoMode;
impl FromStr for CryptoMode {
type Err = UnrecognisedCryptoMode;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"aead_aes256_gcm_rtpsize" => Ok(Self::Aes256Gcm),
"aead_xchacha20_poly1305_rtpsize" => Ok(Self::XChaCha20Poly1305),
_ => Err(UnrecognisedCryptoMode),
}
}
}
impl CryptoMode {
/// Returns the underlying crypto algorithm used by a given [`CryptoMode`].
#[must_use]
pub(crate) const fn algorithm(self) -> EncryptionAlgorithm {
match self {
CryptoMode::Aes256Gcm => EncryptionAlgorithm::Aes256Gcm,
CryptoMode::XChaCha20Poly1305 => EncryptionAlgorithm::XChaCha20Poly1305,
}
}
/// Returns an encryption cipher based on the supplied key.
///
/// Creation fails if the key is the incorrect length for the target cipher.
pub(crate) fn cipher_from_key(self, key: &[u8]) -> Result<Cipher, InvalidLength> {
match self.algorithm() {
EncryptionAlgorithm::Aes256Gcm => Aes256Gcm::new_from_slice(key)
.map(Box::new)
.map(Cipher::Aes256Gcm),
EncryptionAlgorithm::XChaCha20Poly1305 =>
XChaCha20Poly1305::new_from_slice(key).map(Cipher::XChaCha20Poly1305),
}
}
/// Returns a local priority score for a given [`CryptoMode`].
///
/// Higher values are preferred.
#[must_use]
pub(crate) fn priority(self) -> u64 {
match self {
CryptoMode::Aes256Gcm => 1,
CryptoMode::XChaCha20Poly1305 => 0,
}
}
/// Returns the best available crypto mode, given the `modes` offered by the Discord voice server.
///
/// If `preferred` is set and the mode exists in the server's supported algorithms, then that
/// mode will be chosen. Otherwise we select the highest-scoring option which is mutually understood.
pub(crate) fn negotiate<It, T>(
modes: It,
preferred: Option<Self>,
) -> Result<Self, ConnectionError>
where
T: AsRef<str>,
It: IntoIterator<Item = T>,
{
let mut best = None;
for el in modes {
let Ok(el) = CryptoMode::from_str(el.as_ref()) else {
// Unsupported mode. Ignore.
continue;
};
let mut el_priority = el.priority();
if let Some(preferred) = preferred {
if el == preferred {
el_priority = u64::MAX;
}
}
let accept = match best {
None => true,
Some((_, score)) if el_priority > score => true,
_ => false,
};
if accept {
best = Some((el, el_priority));
}
}
best.map(|(v, _)| v)
.ok_or(ConnectionError::CryptoModeUnavailable)
}
/// Returns the name of a mode as it will appear during negotiation.
#[must_use]
pub const fn to_request_str(self) -> &'static str {
match self {
Self::Aes256Gcm => "aead_aes256_gcm_rtpsize",
Self::XChaCha20Poly1305 => "aead_xchacha20_poly1305_rtpsize",
}
}
/// Returns the nonce length in bytes required by algorithm.
#[must_use]
pub const fn algorithm_nonce_size(self) -> usize {
use typenum::Unsigned as _;
match self {
Self::XChaCha20Poly1305 => <XChaCha20Poly1305 as AeadCore>::NonceSize::USIZE, // => 24
Self::Aes256Gcm => <Aes256Gcm as AeadCore>::NonceSize::USIZE, // => 12
}
}
/// Returns the number of bytes each nonce is stored as within
/// a packet.
#[must_use]
pub const fn nonce_size(self) -> usize {
match self {
Self::Aes256Gcm | Self::XChaCha20Poly1305 => 4,
}
}
/// Returns the number of bytes occupied by the encryption scheme
/// which fall before the payload.
///
/// Method name duplicated until v0.5, to prevent breaking change.
#[must_use]
pub(crate) const fn payload_prefix_len(self) -> usize {
match self {
CryptoMode::Aes256Gcm | CryptoMode::XChaCha20Poly1305 => 0,
}
}
/// Returns the tag length in bytes.
#[must_use]
pub(crate) const fn encryption_tag_len(self) -> usize {
self.algorithm().encryption_tag_len()
}
/// Returns the number of bytes occupied by the encryption scheme
/// which fall after the payload.
#[must_use]
pub const fn payload_suffix_len(self) -> usize {
self.nonce_size() + self.encryption_tag_len()
}
/// Returns the number of bytes occupied by an encryption scheme's tag which
/// fall *after* the payload.
#[must_use]
pub const fn tag_suffix_len(self) -> usize {
self.encryption_tag_len()
}
/// Calculates the number of additional bytes required compared
/// to an unencrypted payload.
#[must_use]
pub const fn payload_overhead(self) -> usize {
self.payload_prefix_len() + self.payload_suffix_len()
}
/// Extracts the byte slice in a packet used as the nonce, and the remaining mutable
/// portion of the packet.
fn nonce_slice<'a>(
self,
_header: &'a [u8],
body: &'a mut [u8],
) -> Result<(&'a [u8], &'a mut [u8]), CryptoError> {
match self {
Self::Aes256Gcm | Self::XChaCha20Poly1305 => {
let len = body.len();
if len < self.payload_suffix_len() {
Err(CryptoError)
} else {
let (body_left, nonce_loc) = body.split_at_mut(len - self.nonce_size());
Ok((nonce_loc, body_left))
}
},
}
}
}
/// State used in nonce generation for the encryption variants in [`CryptoMode`].
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
#[non_exhaustive]
pub enum CryptoState {
/// An additional random 4B suffix is used as the source of nonce bytes for the packet.
/// This nonce value increments by `1` with each packet.
///
/// The last used nonce is stored.
Aes256Gcm(Wrapping<u32>),
/// An additional random 4B suffix is used as the source of nonce bytes for the packet.
/// This nonce value increments by `1` with each packet.
///
/// The last used nonce is stored.
XChaCha20Poly1305(Wrapping<u32>),
}
impl From<CryptoMode> for CryptoState {
fn from(val: CryptoMode) -> Self {
match val {
CryptoMode::Aes256Gcm => CryptoState::Aes256Gcm(Wrapping(rand::random::<u32>())),
CryptoMode::XChaCha20Poly1305 =>
CryptoState::XChaCha20Poly1305(Wrapping(rand::random::<u32>())),
}
}
}
impl CryptoState {
/// Writes packet nonce into the body, if required, returning the new length.
pub fn write_packet_nonce(
&mut self,
packet: &mut impl MutablePacket,
payload_end: usize,
) -> usize {
let mode = self.kind();
let endpoint = payload_end + mode.payload_suffix_len();
let startpoint = endpoint - mode.nonce_size();
match self {
Self::Aes256Gcm(ref mut i) | Self::XChaCha20Poly1305(ref mut i) => {
(&mut packet.payload_mut()[startpoint..endpoint])
.write_u32::<NetworkEndian>(i.0)
.expect(
"Nonce size is guaranteed to be sufficient to write u32 for lite tagging.",
);
*i += Wrapping(1);
},
}
endpoint
}
/// Returns the underlying (stateless) type of the active crypto mode.
#[must_use]
pub fn kind(self) -> CryptoMode {
CryptoMode::from(self)
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq, Hash)]
pub(crate) enum EncryptionAlgorithm {
Aes256Gcm,
XChaCha20Poly1305,
}
impl EncryptionAlgorithm {
#[must_use]
pub(crate) const fn encryption_tag_len(self) -> usize {
match self {
Self::Aes256Gcm => <Aes256Gcm as AeadCore>::TagSize::USIZE, // 16
Self::XChaCha20Poly1305 => <XChaCha20Poly1305 as AeadCore>::TagSize::USIZE, // 16
}
}
}
impl From<&Cipher> for EncryptionAlgorithm {
fn from(value: &Cipher) -> Self {
match value {
Cipher::XChaCha20Poly1305(_) => EncryptionAlgorithm::XChaCha20Poly1305,
Cipher::Aes256Gcm(_) => EncryptionAlgorithm::Aes256Gcm,
}
}
}
#[derive(Clone)]
pub enum Cipher {
XChaCha20Poly1305(XChaCha20Poly1305),
Aes256Gcm(Box<Aes256Gcm>),
}
impl Cipher {
#[must_use]
pub(crate) fn mode(&self) -> CryptoMode {
match self {
Cipher::XChaCha20Poly1305(_) => CryptoMode::XChaCha20Poly1305,
Cipher::Aes256Gcm(_) => CryptoMode::Aes256Gcm,
}
}
#[must_use]
pub(crate) fn encryption_tag_len(&self) -> usize {
EncryptionAlgorithm::from(self).encryption_tag_len()
}
/// Encrypts a Discord RT(C)P packet using the given key.
///
/// Use of this requires that the input packet has had a nonce generated in the correct location,
/// and `payload_len` specifies the number of bytes after the header including this nonce.
#[inline]
pub fn encrypt_pkt_in_place(
&self,
packet: &mut impl MutablePacket,
payload_len: usize,
) -> Result<(), CryptoError> {
let mode = self.mode();
let header_len = packet.packet().len() - packet.payload().len();
let (header, body) = packet.packet_mut().split_at_mut(header_len);
let (slice_to_use, body_remaining) = mode.nonce_slice(header, &mut body[..payload_len])?;
let tag_size = self.encryption_tag_len();
// body_remaining is now correctly truncated to exclude the nonce by this point.
// the true_payload to encrypt is within the buf[prefix:-suffix].
let (_, body_remaining) = body_remaining.split_at_mut(mode.payload_prefix_len());
let (body, post_payload) =
body_remaining.split_at_mut(body_remaining.len() - mode.tag_suffix_len());
// All these Nonce types are distinct at the type level
// (96b for AES, 192b for XChaCha).
match self {
// The below variants follow part of the SRTP spec (RFC3711, sec 3.1)
// by requiring that we include the cleartext header portion as
// authenticated data.
Self::Aes256Gcm(aes_gcm) => {
let mut nonce = aes_gcm::Nonce::default();
nonce[..mode.nonce_size()].copy_from_slice(slice_to_use);
let tag = aes_gcm.encrypt_in_place_detached(&nonce, header, body)?;
post_payload[..tag_size].copy_from_slice(&tag[..]);
},
Self::XChaCha20Poly1305(cha_cha_poly1305) => {
let mut nonce = chacha20poly1305::XNonce::default();
nonce[..mode.nonce_size()].copy_from_slice(slice_to_use);
let tag = cha_cha_poly1305.encrypt_in_place_detached(&nonce, header, body)?;
post_payload[..tag_size].copy_from_slice(&tag[..]);
},
}
Ok(())
}
#[cfg(any(feature = "receive", test))]
pub(crate) fn decrypt_rtp_in_place(
&self,
packet: &mut MutableRtpPacket<'_>,
) -> Result<(usize, usize), InternalError> {
// An exciting difference from the SRTP spec: Discord begins encryption
// after the RTP extension *header*, encrypting the extensions themselves,
// whereas the spec leaves all extensions in the clear.
// This header is described as the 'extension preamble'.
let has_extension = packet.get_extension() != 0;
let plain_bytes = if has_extension {
// CSRCs and extension bytes will be in the plaintext segment.
// We will need these demarcated to select the right bytes to
// decrypt, and to use as auth data.
RtpExtensionPacket::minimum_packet_size()
} else {
0
};
let (_, end) = self.decrypt_pkt_in_place(packet, plain_bytes)?;
// Update the start estimate to account for bytes occupied by extension headers.
let payload_offset = if has_extension {
let payload = packet.payload();
let extension =
RtpExtensionPacket::new(payload).ok_or(InternalError::IllegalVoicePacket)?;
extension.packet().len() - extension.payload().len()
} else {
0
};
Ok((payload_offset, end))
}
#[cfg(feature = "receive")]
pub(crate) fn decrypt_rtcp_in_place(
&self,
packet: &mut MutableRtcpPacket<'_>,
) -> Result<(usize, usize), InternalError> {
// RTCP/SRTCP have identical handling -- no var-length elements
// are included as part of the plaintext.
self.decrypt_pkt_in_place(packet, 0)
}
/// Decrypts an arbitrary packet using the given key.
///
/// If successful, this returns the number of bytes to be ignored from the
/// start and end of the packet payload.
#[inline]
#[cfg(any(feature = "receive", test))]
pub(crate) fn decrypt_pkt_in_place(
&self,
packet: &mut impl MutablePacket,
n_plaintext_body_bytes: usize,
) -> Result<(usize, usize), InternalError> {
let mode = self.mode();
let header_len = packet.packet().len() - packet.payload().len();
let plaintext_end = header_len + n_plaintext_body_bytes;
let (plaintext, ciphertext) = packet
.packet_mut()
.split_at_mut_checked(plaintext_end)
.ok_or(CryptoError)?;
let (slice_to_use, body_remaining) = mode.nonce_slice(plaintext, ciphertext)?;
let (pre_payload, body_remaining) = body_remaining
.split_at_mut_checked(mode.payload_prefix_len())
.ok_or(CryptoError)?;
let suffix_split_point = body_remaining
.len()
.checked_sub(mode.tag_suffix_len())
.ok_or(CryptoError)?;
let (body, post_payload) = body_remaining
.split_at_mut_checked(suffix_split_point)
.ok_or(CryptoError)?;
let tag_size = self.encryption_tag_len();
match self {
// The below variants follow part of the SRTP spec (RFC3711, sec 3.1)
// by requiring that we include the cleartext header portion as
// authenticated data.
Self::Aes256Gcm(aes_gcm) => {
let mut nonce = aes_gcm::Nonce::default();
nonce[..mode.nonce_size()].copy_from_slice(slice_to_use);
let tag = aes_gcm::Tag::from_slice(&post_payload[..tag_size]);
aes_gcm.decrypt_in_place_detached(&nonce, plaintext, body, tag)?;
},
Self::XChaCha20Poly1305(cha_cha_poly1305) => {
let mut nonce = chacha20poly1305::XNonce::default();
nonce[..mode.nonce_size()].copy_from_slice(slice_to_use);
let tag = chacha20poly1305::Tag::from_slice(&post_payload[..tag_size]);
cha_cha_poly1305.decrypt_in_place_detached(&nonce, plaintext, body, tag)?;
},
}
Ok((
plaintext_end + pre_payload.len(),
post_payload.len() + slice_to_use.len(),
))
}
}
#[cfg(test)]
mod test {
use super::*;
use discortp::rtp::MutableRtpPacket;
#[test]
fn small_packet_decrypts_error() {
let mut buf = [0u8; MutableRtpPacket::minimum_packet_size()];
let modes = [CryptoMode::Aes256Gcm, CryptoMode::XChaCha20Poly1305];
let mut pkt = MutableRtpPacket::new(&mut buf[..]).unwrap();
for mode in modes {
// Coincidentally, these are all 32B for now.
let cipher = mode.cipher_from_key(&[1u8; 32]).unwrap();
// AIM: should error, and not panic.
assert!(cipher.decrypt_rtp_in_place(&mut pkt).is_err());
}
}
#[test]
fn symmetric_encrypt_decrypt_tag_after_data() {
const TRUE_PAYLOAD: [u8; 8] = [1, 2, 3, 4, 5, 6, 7, 8];
for mode in [CryptoMode::Aes256Gcm, CryptoMode::XChaCha20Poly1305] {
let mut buf = vec![
0u8;
MutableRtpPacket::minimum_packet_size()
+ TRUE_PAYLOAD.len()
+ mode.nonce_size()
+ mode.encryption_tag_len()
];
buf.fill(0);
let cipher = mode.cipher_from_key(&[7u8; 32]).unwrap();
let mut pkt = MutableRtpPacket::new(&mut buf[..]).unwrap();
let mut crypto_state = CryptoState::from(mode);
let payload = pkt.payload_mut();
payload[mode.payload_prefix_len()..TRUE_PAYLOAD.len()].copy_from_slice(&TRUE_PAYLOAD);
let final_payload_size = crypto_state.write_packet_nonce(&mut pkt, TRUE_PAYLOAD.len());
let enc_succ = cipher.encrypt_pkt_in_place(&mut pkt, final_payload_size);
assert!(enc_succ.is_ok());
let final_pkt_len = MutableRtpPacket::minimum_packet_size() + final_payload_size;
let mut pkt = MutableRtpPacket::new(&mut buf[..final_pkt_len]).unwrap();
assert!(cipher.decrypt_rtp_in_place(&mut pkt).is_ok());
}
}
#[test]
fn negotiate_cryptomode() {
// If we have no preference (or our preference is missing), choose the highest available in the set.
let test_set =
[CryptoMode::XChaCha20Poly1305, CryptoMode::Aes256Gcm].map(CryptoMode::to_request_str);
assert_eq!(
CryptoMode::negotiate(test_set, None).unwrap(),
CryptoMode::Aes256Gcm
);
let test_set_missing = [CryptoMode::XChaCha20Poly1305].map(CryptoMode::to_request_str);
assert_eq!(
CryptoMode::negotiate(test_set_missing, Some(CryptoMode::Aes256Gcm)).unwrap(),
CryptoMode::XChaCha20Poly1305
);
// Preference wins in spite of the defined `priority` value.
assert_eq!(
CryptoMode::negotiate(test_set, Some(CryptoMode::XChaCha20Poly1305)).unwrap(),
CryptoMode::XChaCha20Poly1305
);
// If there is no mutual intelligibility, return an error.
let bad_modes = ["not_real", "des", "rc5"];
assert!(CryptoMode::negotiate(bad_modes, None).is_err());
assert!(CryptoMode::negotiate(bad_modes, Some(CryptoMode::Aes256Gcm)).is_err());
}
}

View File

@@ -0,0 +1,125 @@
use opus2::Channels as OpusChannels;
/// Decode behaviour for received RTP packets within the driver.
#[derive(Clone, Copy, Debug, Eq, PartialEq, Hash)]
#[non_exhaustive]
pub enum DecodeMode {
/// Packets received from Discord are handed over to events without any
/// changes applied.
///
/// No CPU work involved.
Pass,
/// Decrypts the body of each received packet.
///
/// Small per-packet CPU use.
Decrypt,
/// Decrypts and decodes each received packet, correctly accounting for losses.
///
/// Larger per-packet CPU use.
Decode(DecodeConfig),
}
impl DecodeMode {
/// Returns whether this mode will decrypt and decode received packets.
#[must_use]
pub fn should_decode(self) -> bool {
matches!(self, DecodeMode::Decode(..))
}
/// Returns whether this mode will decrypt received packets.
#[must_use]
pub fn should_decrypt(self) -> bool {
self != DecodeMode::Pass
}
}
/// Configuration for [`DecodeMode::Decode`]
#[non_exhaustive]
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq, Hash)]
pub struct DecodeConfig {
/// Configures the channel layout for output audio.
///
/// Defaults to [`Channels::Stereo`].
pub channels: Channels,
/// Configures the sample rate for output audio.
///
/// Defaults to [`SampleRate::Hz48000`].
pub sample_rate: SampleRate,
}
impl DecodeConfig {
/// Creates a new [`DecodeConfig`] with the specified channels and sample rate.
#[must_use]
pub fn new(channels: Channels, sample_rate: SampleRate) -> Self {
Self {
channels,
sample_rate,
}
}
}
/// The channel layout of output audio when using [`DecodeMode::Decode`].
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq, Hash)]
#[non_exhaustive]
pub enum Channels {
/// Decode received audio packets into a single channel.
Mono,
/// Decode received audio packets into two interleaved channels.
///
/// Received mono packets' samples will automatically be duplicated across
/// both channels.
///
/// The default choice.
#[default]
Stereo,
}
impl Channels {
pub(crate) fn channels(self) -> usize {
match self {
Channels::Mono => 1,
Channels::Stereo => 2,
}
}
}
impl From<Channels> for OpusChannels {
fn from(value: Channels) -> Self {
match value {
Channels::Mono => OpusChannels::Mono,
Channels::Stereo => OpusChannels::Stereo,
}
}
}
/// The sample rate of output audio when using [`DecodeMode::Decode`].
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq, Hash)]
#[non_exhaustive]
pub enum SampleRate {
/// Decode to a sample rate of 8kHz.
Hz8000,
/// Decode to a sample rate of 12kHz.
Hz12000,
/// Decode to a sample rate of 16kHz.
Hz16000,
/// Decode to a sample rate of 24kHz.
Hz24000,
/// Decode to a sample rate of 48kHz.
///
/// The preferred option for encoding/decoding at or above CD quality.
#[default]
Hz48000,
}
impl From<SampleRate> for u32 {
fn from(value: SampleRate) -> Self {
match value {
SampleRate::Hz8000 => 8000,
SampleRate::Hz12000 => 12000,
SampleRate::Hz16000 => 16000,
SampleRate::Hz24000 => 24000,
SampleRate::Hz48000 => 48000,
}
}
}

View File

@@ -0,0 +1,59 @@
use opus2::Channels;
use symphonia_core::audio::Layout;
use crate::constants::{MONO_FRAME_SIZE, STEREO_FRAME_SIZE};
/// Mixing behaviour for sent audio sources processed within the driver.
///
/// This has no impact on Opus packet passthrough, which will pass packets
/// irrespective of their channel count.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum MixMode {
/// Audio sources will be downmixed into a mono buffer.
Mono,
/// Audio sources will be mixed into into a stereo buffer, where mono sources
/// will be duplicated into both channels.
Stereo,
}
impl MixMode {
pub(crate) const fn to_opus(self) -> Channels {
match self {
Self::Mono => Channels::Mono,
Self::Stereo => Channels::Stereo,
}
}
pub(crate) const fn sample_count_in_frame(self) -> usize {
match self {
Self::Mono => MONO_FRAME_SIZE,
Self::Stereo => STEREO_FRAME_SIZE,
}
}
pub(crate) const fn channels(self) -> usize {
match self {
Self::Mono => 1,
Self::Stereo => 2,
}
}
pub(crate) const fn symph_layout(self) -> Layout {
match self {
Self::Mono => Layout::Mono,
Self::Stereo => Layout::Stereo,
}
}
}
impl From<MixMode> for Layout {
fn from(val: MixMode) -> Self {
val.symph_layout()
}
}
impl From<MixMode> for Channels {
fn from(val: MixMode) -> Self {
val.to_opus()
}
}

357
app/vendor/songbird/src/driver/mod.rs vendored Normal file
View File

@@ -0,0 +1,357 @@
//! Runner for a voice connection.
//!
//! Songbird's driver is a mixed-sync system, using:
//! * Asynchronous connection management, event-handling, and gateway integration.
//! * Synchronous audio mixing, packet generation, and encoding.
//!
//! This splits up work according to its IO/compute bound nature, preventing packet
//! generation from being slowed down past its deadline, or from affecting other
//! asynchronous tasks your bot must handle.
#[cfg(feature = "internals")]
pub mod bench_internals;
pub(crate) mod connection;
mod crypto;
#[cfg(feature = "receive")]
mod decode_mode;
mod mix_mode;
pub mod retry;
mod scheduler;
pub(crate) mod tasks;
#[cfg(test)]
pub(crate) mod test_config;
#[cfg(any(test, feature = "internals"))]
mod test_impls;
use connection::error::{Error, Result};
pub use crypto::CryptoMode;
pub(crate) use crypto::CryptoState;
#[cfg(feature = "receive")]
pub use decode_mode::*;
pub use mix_mode::MixMode;
pub use scheduler::{
get_default_scheduler,
Config as SchedulerConfig,
Error as SchedulerError,
LiveStatBlock,
Mode as SchedulerMode,
Scheduler,
};
#[cfg(test)]
pub use test_config::*;
#[cfg(any(test, feature = "internals"))]
pub use test_impls::*;
#[cfg(feature = "builtin-queue")]
use crate::tracks::TrackQueue;
use crate::{
events::EventData,
input::Input,
tracks::{Track, TrackHandle},
Config,
ConnectionInfo,
Event,
EventHandler,
};
use core::{
future::Future,
pin::Pin,
task::{Context, Poll},
};
use flume::{r#async::RecvFut, SendError, Sender};
/// Opus encoder bitrate settings.
pub use opus2::{self as opus, Bitrate};
#[cfg(feature = "builtin-queue")]
use std::time::Duration;
#[allow(unused_imports)]
pub use tasks::disposal::DisposalThread;
use tasks::message::CoreMessage;
use tracing::instrument;
/// The control object for a Discord voice connection, handling connection,
/// mixing, encoding, en/decryption, and event generation.
///
/// When compiled with the `"builtin-queue"` feature, each driver includes a track queue
/// as a convenience to prevent the additional overhead of per-guild state management.
#[derive(Clone, Debug)]
pub struct Driver {
config: Config,
self_mute: bool,
sender: Sender<CoreMessage>,
// Making this an Option is an abhorrent hack to coerce the borrow checker
// into letting us have an &TrackQueue at the same time as an &mut Driver.
// This is probably preferable to cloning the driver: Arc<...> should be nonzero
// and if the compiler's smart we'll just codegen a pointer swap. It definitely makes
// use of NonZero.
#[cfg(feature = "builtin-queue")]
queue: Option<TrackQueue>,
}
impl Driver {
/// Creates a new voice driver.
///
/// This will create the core voice tasks in the background.
#[inline]
#[must_use]
pub fn new(config: Config) -> Self {
let sender = Self::start_inner(config.clone());
Driver {
config,
self_mute: false,
sender,
#[cfg(feature = "builtin-queue")]
queue: Some(TrackQueue::default()),
}
}
fn start_inner(config: Config) -> Sender<CoreMessage> {
let (tx, rx) = flume::unbounded();
tasks::start(config, rx, tx.clone());
tx
}
fn restart_inner(&mut self) {
self.sender = Self::start_inner(self.config.clone());
self.mute(self.self_mute);
}
/// Connects to a voice channel using the specified server.
///
/// This method instantly contacts the driver tasks, and its
/// does not need to be `await`ed to start the actual connection.
#[instrument(skip(self))]
pub fn connect(&mut self, info: ConnectionInfo) -> Connect {
let (tx, rx) = flume::bounded(1);
self.raw_connect(info, tx);
Connect {
inner: rx.into_recv_async(),
}
}
/// Connects to a voice channel using the specified server.
#[instrument(skip(self))]
pub(crate) fn raw_connect(&mut self, info: ConnectionInfo, tx: Sender<Result<()>>) {
self.send(CoreMessage::ConnectWithResult(info, tx));
}
/// Leaves the current voice channel, disconnecting from it.
///
/// This does *not* forget settings, like whether to be self-deafened or
/// self-muted.
#[instrument(skip(self))]
pub fn leave(&mut self) {
self.send(CoreMessage::Disconnect);
}
/// Sets whether the current connection is to be muted.
///
/// If there is no live voice connection, then this only acts as a settings
/// update for future connections.
#[instrument(skip(self))]
pub fn mute(&mut self, mute: bool) {
self.self_mute = mute;
self.send(CoreMessage::Mute(mute));
}
/// Returns whether the driver is muted (i.e., processes audio internally
/// but submits none).
#[instrument(skip(self))]
pub fn is_mute(&self) -> bool {
self.self_mute
}
/// Plays audio from an input, returning a handle for further control.
#[instrument(skip(self, input))]
pub fn play_input(&mut self, input: Input) -> TrackHandle {
self.play(input.into())
}
/// Plays audio from an input, returning a handle for further control.
///
/// Unlike [`Self::play_input`], this stops all other inputs attached
/// to the channel.
#[instrument(skip(self, input))]
pub fn play_only_input(&mut self, input: Input) -> TrackHandle {
self.play_only(input.into())
}
/// Plays audio from a [`Track`] object.
///
/// The main difference between this function and [`Self::play_input`] is
/// that this allows for direct manipulation of the [`Track`] object
/// before it is passed over to the voice and mixing contexts.
#[instrument(skip(self, track))]
pub fn play(&mut self, track: Track) -> TrackHandle {
let (handle, ctx) = track.into_context();
self.send(CoreMessage::AddTrack(Box::new(ctx)));
handle
}
/// Exclusively plays audio from a [`Track`] object.
///
/// As in [`Self::play_only_input`], this stops all other sources attached to the
/// channel. Like [`Self::play`], however, this allows for direct manipulation of the
/// [`Track`] object before it is passed over to the voice and mixing contexts.
#[instrument(skip(self, track))]
pub fn play_only(&mut self, track: Track) -> TrackHandle {
let (handle, ctx) = track.into_context();
self.send(CoreMessage::SetTrack(Some(Box::new(ctx))));
handle
}
/// Sets the bitrate for encoding Opus packets sent along
/// the channel being managed.
///
/// The default rate is 128 kbps.
/// Sensible values range between `Bits(512)` and `Bits(512_000)`
/// bits per second.
/// Alternatively, `Auto` and `Max` remain available.
#[instrument(skip(self))]
pub fn set_bitrate(&mut self, bitrate: Bitrate) {
self.send(CoreMessage::SetBitrate(bitrate));
}
/// Stops playing audio from all sources, if any are set.
#[instrument(skip(self))]
pub fn stop(&mut self) {
self.send(CoreMessage::SetTrack(None));
}
/// Sets the configuration for this driver (and parent `Call`, if applicable).
#[instrument(skip(self))]
pub fn set_config(&mut self, config: Config) {
self.config = config.clone();
self.send(CoreMessage::SetConfig(config));
}
/// Returns a view of this driver's configuration.
#[instrument(skip(self))]
pub fn config(&self) -> &Config {
&self.config
}
/// Attach a global event handler to an audio context. Global events may receive
/// any [`EventContext`].
///
/// Global timing events will tick regardless of whether audio is playing,
/// so long as the bot is connected to a voice channel, and have no tracks.
/// [`TrackEvent`]s will respond to all relevant tracks, giving some audio elements.
///
/// Users **must** ensure that no costly work or blocking occurs
/// within the supplied function or closure. *Taking excess time could prevent
/// timely sending of packets, causing audio glitches and delays*.
///
/// [`TrackEvent`]: crate::events::TrackEvent
/// [`EventContext`]: crate::events::EventContext
#[instrument(skip(self, action))]
pub fn add_global_event<F: EventHandler + 'static>(&mut self, event: Event, action: F) {
self.send(CoreMessage::AddEvent(EventData::new(event, action)));
}
/// Removes all global event handlers from an audio context.
#[instrument(skip(self))]
pub fn remove_all_global_events(&mut self) {
self.send(CoreMessage::RemoveGlobalEvents);
}
/// Sends a message to the inner tasks, restarting it if necessary.
fn send(&mut self, status: CoreMessage) {
// Restart thread if it errored.
if let Err(SendError(status)) = self.sender.send(status) {
self.restart_inner();
self.sender.send(status).unwrap();
}
}
}
#[cfg(feature = "builtin-queue")]
impl Driver {
/// Returns a reference to this driver's built-in queue.
///
/// Requires the `"builtin-queue"` feature.
/// Queue additions should be made via [`Driver::enqueue`] and
/// [`Driver::enqueue_input`].
#[must_use]
pub fn queue(&self) -> &TrackQueue {
self.queue
.as_ref()
.expect("Queue: The only case this can fail is if a previous queue operation panicked.")
}
/// Adds an audio [`Input`] to this driver's built-in queue.
///
/// Requires the `"builtin-queue"` feature.
pub async fn enqueue_input(&mut self, input: Input) -> TrackHandle {
self.enqueue(input.into()).await
}
/// Adds an existing [`Track`] to this driver's built-in queue.
///
/// Requires the `"builtin-queue"` feature.
pub async fn enqueue(&mut self, mut track: Track) -> TrackHandle {
let preload_time = TrackQueue::get_preload_time(&mut track).await;
self.enqueue_with_preload(track, preload_time)
}
/// Add an existing [`Track`] to the queue, using a known time to preload the next track.
///
/// See [`TrackQueue::add_with_preload`] for how `preload_time` is used.
///
/// Requires the `"builtin-queue"` feature.
pub fn enqueue_with_preload(
&mut self,
track: Track,
preload_time: Option<Duration>,
) -> TrackHandle {
let queue = self.queue.take().expect(
"Enqueue: The only case this can fail is if a previous queue operation panicked.",
);
let handle = queue.add_with_preload(track, self, preload_time);
self.queue = Some(queue);
handle
}
}
impl Default for Driver {
fn default() -> Self {
Self::new(Config::default())
}
}
impl Drop for Driver {
/// Leaves the current connected voice channel, if connected to one, and
/// forgets all configurations relevant to this Handler.
fn drop(&mut self) {
drop(self.sender.send(CoreMessage::Poison));
}
}
/// Future for a call to [`Driver::connect`].
///
/// This future awaits the *result* of a connection; the driver
/// is messaged at the time of the call.
pub struct Connect {
inner: RecvFut<'static, Result<()>>,
}
impl Future for Connect {
type Output = Result<()>;
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
match Pin::new(&mut self.inner).poll(cx) {
Poll::Ready(r) => Poll::Ready(r.map_err(|_| Error::AttemptDiscarded).and_then(|x| x)),
Poll::Pending => Poll::Pending,
}
}
}

View File

@@ -0,0 +1,51 @@
//! Configuration for connection retries.
mod strategy;
pub use self::strategy::*;
use std::num::NonZeroU8;
use crate::FloatDuration;
/// Configuration to be used for retrying driver connection attempts.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Retry {
/// Strategy used to determine how long to wait between retry attempts.
///
/// *Defaults to an [`ExponentialBackoff`] from 0.25s
/// to 10s, with a jitter of `0.1`.*
///
/// [`ExponentialBackoff`]: Strategy::Backoff
pub strategy: Strategy,
/// The maximum number of retries to attempt.
///
/// `None` will attempt an infinite number of retries,
/// while `Some(0)` will attempt to connect *once* (no retries).
///
/// *Defaults to `Some(5)`.*
pub retry_limit: Option<NonZeroU8>,
}
impl Default for Retry {
fn default() -> Self {
Self {
strategy: Strategy::Backoff(ExponentialBackoff::default()),
retry_limit: Some(const { NonZeroU8::new(5).unwrap() }),
}
}
}
impl Retry {
pub(crate) fn retry_in(
&self,
last_wait: Option<FloatDuration>,
attempts: u8,
) -> Option<FloatDuration> {
if self.retry_limit.is_none_or(|a| attempts < a.get()) {
Some(self.strategy.retry_in(last_wait))
} else {
None
}
}
}

View File

@@ -0,0 +1,77 @@
use std::time::Duration;
use rand::random;
use crate::FloatDuration;
/// Logic used to determine how long to wait between retry attempts.
#[derive(Clone, Copy, Debug, PartialEq)]
#[non_exhaustive]
pub enum Strategy {
/// The driver will wait for the same amount of time between each retry.
Every(FloatDuration),
/// Exponential backoff waiting strategy, where the duration between
/// attempts (approximately) doubles each time.
Backoff(ExponentialBackoff),
}
impl Strategy {
pub(crate) fn retry_in(&self, last_wait: Option<FloatDuration>) -> FloatDuration {
match self {
Self::Every(t) => *t,
Self::Backoff(exp) => exp.retry_in(last_wait),
}
}
}
/// Exponential backoff waiting strategy.
///
/// Each attempt waits for twice the last delay plus/minus a
/// random jitter, clamped to a min and max value.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct ExponentialBackoff {
/// Minimum amount of time to wait between retries.
///
/// *Defaults to 0.25s.*
pub min: FloatDuration,
/// Maximum amount of time to wait between retries.
///
/// This will be clamped to `>=` min.
///
/// *Defaults to 10s.*
pub max: FloatDuration,
/// Amount of uniform random jitter to apply to generated wait times.
/// I.e., 0.1 will add +/-10% to generated intervals.
///
/// This is restricted to within +/-100%.
///
/// *Defaults to `0.1`.*
pub jitter: f32,
}
impl Default for ExponentialBackoff {
fn default() -> Self {
Self {
min: Duration::from_millis(250).into(),
max: Duration::from_secs(10).into(),
jitter: 0.1,
}
}
}
impl ExponentialBackoff {
pub(crate) fn retry_in(&self, last_wait: Option<FloatDuration>) -> FloatDuration {
let min = self.min.as_secs_f32();
let max = self.max.as_secs_f32();
let attempt = last_wait.map_or(min, |t| 2.0 * t.as_secs_f32());
let perturb = (1.0 - (self.jitter * 2.0 * (random::<f32>() - 1.0))).clamp(0.0, 2.0);
let mut target_time = attempt * perturb;
// Now clamp target time into given range.
let safe_max = if max < min { min } else { max };
target_time = target_time.clamp(min, safe_max);
FloatDuration::from_secs_f32(target_time)
}
}

View File

@@ -0,0 +1,64 @@
use super::*;
/// Configuration for how a [`Scheduler`] handles tasks.
///
/// [`Scheduler`]: super::Scheduler
#[derive(Clone, Debug)]
#[non_exhaustive]
pub struct Config {
/// How Live mixer tasks will be mapped to individual threads.
///
/// Defaults to `Mode::MaxPerThread(16)`.
pub strategy: Mode,
/// Move costly mixers to another thread if their parent worker is at
/// risk of missing its deadlines.
///
/// Defaults to `true`.
pub move_expensive_tasks: bool,
}
impl Default for Config {
fn default() -> Self {
Self {
strategy: Mode::default(),
move_expensive_tasks: true,
}
}
}
/// Strategies for mapping live mixer tasks to individual threads.
///
/// Defaults to `MaxPerThread(16)`.
#[derive(Clone, Debug)]
#[non_exhaustive]
pub enum Mode {
/// Allows at most `n` tasks to run per thread.
MaxPerThread(NonZeroUsize),
}
impl Mode {
/// Returns the number of `Mixer`s that a scheduler should preallocate
/// resources for.
pub(crate) fn prealloc_size(&self) -> usize {
match self {
Self::MaxPerThread(n) => n.get(),
}
}
/// Returns the maximum number of concurrent mixers that a scheduler is
/// allowed to place on a single thread.
///
/// Future scheduling modes may choose to limit *only* on execution cost.
#[allow(clippy::unnecessary_wraps)]
pub(crate) fn task_limit(&self) -> Option<usize> {
match self {
Self::MaxPerThread(n) => Some(n.get()),
}
}
}
impl Default for Mode {
fn default() -> Self {
Self::MaxPerThread(DEFAULT_MIXERS_PER_THREAD)
}
}

View File

@@ -0,0 +1,335 @@
use std::{collections::HashMap, time::Duration};
use nohash_hasher::{BuildNoHashHasher, IntMap};
use tokio::time::{Instant as TokInstant, Interval};
use tracing::info;
use crate::constants::*;
use super::*;
const THREAD_CULL_TIMER: Duration = Duration::from_secs(60);
/// An async task responsible for maintaining UDP keepalives and event state for inactive
/// `Mixer` tasks.
pub(crate) struct Idle {
config: Config,
cull_timer: Duration,
tasks: IntMap<TaskId, ParkedMixer>,
// track taskids which are live to prevent their realloc? unlikely w u64 but still
pub(crate) stats: Arc<StatBlock>,
rx: Receiver<SchedulerMessage>,
tx: Sender<SchedulerMessage>,
next_id: TaskId,
next_worker_id: WorkerId,
workers: Vec<Worker>,
to_cull: Vec<TaskId>,
}
impl Idle {
pub fn new(config: Config) -> (Self, Sender<SchedulerMessage>) {
let (tx, rx) = flume::unbounded();
let stats = Arc::default();
let tasks = HashMap::with_capacity_and_hasher(128, BuildNoHashHasher::default());
// TODO: include heap of keepalive sending times?
let out = Self {
config,
cull_timer: THREAD_CULL_TIMER,
tasks,
stats,
rx,
tx: tx.clone(),
next_id: TaskId::new(),
next_worker_id: WorkerId::new(),
workers: Vec::with_capacity(16),
to_cull: vec![],
};
(out, tx)
}
/// Run the inner task until all external `Scheduler` handles are dropped.
async fn run(&mut self) {
let mut interval = tokio::time::interval(TIMESTEP_LENGTH);
while self.run_once(&mut interval).await {}
}
/// Run one 'tick' of idle thread maintenance.
///
/// This is a priority system over 2 main tasks:
/// 1) handle scheduling/upgrade/action requests for mixers
/// 2) [every 20ms]tick the main timer for each task, send keepalive if
/// needed, reclaim & cull workers.
///
/// Idle mixers spawn an async task each to forward their `MixerMessage`s
/// on to this task to be handled by 1). These tasks self-terminate if a
/// message would make a mixer `now_live`.
async fn run_once(&mut self, interval: &mut Interval) -> bool {
tokio::select! {
biased;
msg = self.rx.recv_async() => match msg {
Ok(SchedulerMessage::NewMixer(rx, ic, cfg)) => {
let mut mixer = ParkedMixer::new(rx, ic, cfg);
let id = self.next_id.incr();
mixer.spawn_forwarder(self.tx.clone(), id);
self.tasks.insert(id, mixer);
self.stats.add_idle_mixer();
},
Ok(SchedulerMessage::Demote(id, mut task)) => {
task.send_gateway_not_speaking();
task.spawn_forwarder(self.tx.clone(), id);
self.tasks.insert(id, task);
},
Ok(SchedulerMessage::Do(id, mix_msg)) => {
let maybe_live = mix_msg.is_mixer_maybe_live();
if let Some(task) = self.tasks.get_mut(&id) {
match task.handle_message(mix_msg) {
Ok(false) if maybe_live => {
if task.mixer.tracks.is_empty() {
// No tracks, likely due to SetConn.
// Recreate message forwarding task.
task.spawn_forwarder(self.tx.clone(), id);
} else {
let task = self.tasks.remove(&id).unwrap();
self.schedule_mixer(task, id, None);
}
},
Ok(false) => {},
Ok(true) | Err(()) => self.to_cull.push(id),
}
} else {
info!("Received post-cull message for {id:?}, discarding.");
}
},
Ok(SchedulerMessage::Overspill(worker_id, id, task)) => {
self.schedule_mixer(task, id, Some(worker_id));
},
Ok(SchedulerMessage::GetStats(tx)) => {
_ = tx.send(self.workers.iter().map(Worker::stats).collect());
},
Ok(SchedulerMessage::Kill) | Err(_) => {
return false;
},
},
_ = interval.tick() => {
// TODO: store keepalive sends in another data structure so
// we don't check every task every 20ms.
//
// if we can also make tick handling lazy(er), we can also optimise for that.
let now = TokInstant::now();
for (id, task) in &mut self.tasks {
// NOTE: this is a non-blocking send so safe from async context.
if task.tick_and_keepalive(now.into()).is_err() {
self.to_cull.push(*id);
}
}
let mut i = 0;
while i < self.workers.len() {
if let Some(then) = self.workers[i].try_mark_empty(now) {
if now.duration_since(then) >= self.cull_timer {
self.workers.swap_remove(i);
continue;
}
}
i += 1;
}
},
}
for id in self.to_cull.drain(..) {
if let Some(tx) = self.tasks.remove(&id).and_then(|t| t.cull_handle) {
_ = tx.send_async(()).await;
}
}
true
}
/// Promote a task to a live mixer thread.
fn schedule_mixer(&mut self, mut task: ParkedMixer, id: TaskId, avoid: Option<WorkerId>) {
if task.send_gateway_speaking().is_ok() {
// If a worker ever completely fails, then we need to remove it here
// `fetch_worker` will either find another, or generate us a new one if
// none exist.
// We need to track ownership of the task coming back via SendError using this
// Option.
let mut loop_task = Some(task);
loop {
let task = loop_task.take().unwrap();
let (worker, idx) = self.fetch_worker(&task, avoid);
match worker.schedule_mixer(id, task) {
Ok(()) => {
self.stats.move_mixer_to_live();
break;
},
Err(e) => {
loop_task = Some(e.0 .1);
let worker = self.workers.swap_remove(idx);
// NOTE: we have incremented worker's live counter for this mixer in
// `schedule_mixer`.
// The only time this branch is ever hit is if a worker crashed, so we
// need to replicate some of their cleanup.
self.stats
.remove_live_mixers(worker.stats().live_mixers().saturating_sub(1));
self.stats.remove_worker();
},
}
}
}
}
/// Fetch the first `Worker` that has room for a new task, creating one if needed.
///
/// If an inbound task has spilled from another thread, then do not reschedule it there.
fn fetch_worker(
&mut self,
task: &ParkedMixer,
avoid: Option<WorkerId>,
) -> (&mut Worker, usize) {
let idx = self
.workers
.iter()
.position(|w| w.can_schedule(task, avoid))
.unwrap_or_else(|| {
self.workers.push(Worker::new(
self.next_worker_id.incr(),
self.config.clone(),
self.tx.clone(),
self.stats.clone(),
));
self.stats.add_worker();
self.workers.len() - 1
});
(&mut self.workers[idx], idx)
}
pub fn spawn(mut self) {
tokio::spawn(async move { self.run().await });
}
}
#[cfg(test)]
mod test {
use super::*;
use crate::{
constants::test_data::FILE_WEBM_TARGET,
driver::{tasks::mixer::Mixer, OutputMode},
input::File,
Driver,
};
use tokio::runtime::Handle;
#[tokio::test]
async fn inactive_mixers_dont_need_threads() {
let sched = Scheduler::new(Config::default());
let cfg = DriverConfig::default().scheduler(sched.clone());
let _drivers: Vec<Driver> = (0..1024).map(|_| Driver::new(cfg.clone())).collect();
tokio::time::sleep(Duration::from_secs(1)).await;
assert_eq!(sched.total_tasks(), 1024);
assert_eq!(sched.live_tasks(), 0);
assert_eq!(sched.worker_threads(), 0);
}
#[tokio::test]
async fn active_mixers_spawn_threads() {
let config = Config {
strategy: Mode::default(),
move_expensive_tasks: false,
};
let sched = Scheduler::new(config);
let (pkt_tx, _pkt_rx) = flume::unbounded();
let cfg = DriverConfig::default()
.scheduler(sched.clone())
.override_connection(Some(OutputMode::Rtp(pkt_tx)));
let n_tasks = 1024;
let _drivers: Vec<Driver> = (0..n_tasks)
.map(|_| {
let mut driver = Driver::new(cfg.clone());
let file = File::new(FILE_WEBM_TARGET);
driver.play_input(file.into());
driver
})
.collect();
tokio::time::sleep(Duration::from_secs(10)).await;
assert_eq!(sched.total_tasks(), n_tasks);
assert_eq!(sched.live_tasks(), n_tasks);
assert_eq!(
sched.worker_threads(),
n_tasks / (DEFAULT_MIXERS_PER_THREAD.get() as u64)
);
}
#[tokio::test]
async fn excess_threads_are_cleaned_up() {
const TEST_TIMER: Duration = Duration::from_millis(500);
let config = Config {
strategy: Mode::MaxPerThread(1.try_into().unwrap()),
move_expensive_tasks: true,
};
let (mut core, tx) = Idle::new(config.clone());
core.cull_timer = TEST_TIMER;
let mut next_id = TaskId::new();
let mut thread_id = WorkerId::new();
let mut handles = vec![];
for i in 0..2 {
let mut worker = Worker::new(
thread_id.incr(),
config.clone(),
tx.clone(),
core.stats.clone(),
);
let ((mixer, listeners), track_handle) =
Mixer::test_with_float_unending(Handle::current(), false);
let send_mixer = ParkedMixer {
mixer: Box::new(mixer),
ssrc: i,
rtp_sequence: i as u16,
rtp_timestamp: i,
park_time: TokInstant::now().into(),
last_cost: None,
cull_handle: None,
};
core.stats.add_idle_mixer();
core.stats.move_mixer_to_live();
worker.schedule_mixer(next_id.incr(), send_mixer).unwrap();
handles.push((track_handle, listeners));
core.workers.push(worker);
}
let mut timer = tokio::time::interval(TIMESTEP_LENGTH);
assert!(core.run_once(&mut timer).await);
// Stop one of the handles, allow it to exit, and then run core again.
handles[1].0.stop().unwrap();
while core.workers[1].is_busy() {
assert!(core.run_once(&mut timer).await);
}
tokio::time::sleep(TEST_TIMER + Duration::from_secs(1)).await;
while core.workers.len() != 1 {
assert!(core.run_once(&mut timer).await);
}
assert_eq!(core.stats.worker_threads(), 0);
}
}

View File

@@ -0,0 +1,823 @@
use std::time::{Duration, Instant};
use discortp::rtp::{MutableRtpPacket, RtpPacket};
use flume::{SendError, TryRecvError};
use tokio::time::Instant as TokInstant;
use crate::{
constants::*,
driver::tasks::{error::Error as DriverError, mixer::Mixer},
};
#[cfg(test)]
use crate::driver::test_config::TickStyle;
use super::*;
/// The send-half of a worker thread, with bookkeeping mechanisms to help
/// the idle task schedule incoming tasks.
pub struct Worker {
id: WorkerId,
stats: Arc<LiveStatBlock>,
config: Config,
tx: Sender<(TaskId, ParkedMixer)>,
known_empty_since: Option<TokInstant>,
}
#[allow(missing_docs)]
impl Worker {
pub fn new(
id: WorkerId,
config: Config,
sched_tx: Sender<SchedulerMessage>,
global_stats: Arc<StatBlock>,
) -> Self {
let stats = Arc::new(LiveStatBlock::default());
let (live_tx, live_rx) = flume::unbounded();
let core = Live::new(
id,
config.clone(),
global_stats,
stats.clone(),
live_rx,
sched_tx,
);
core.spawn();
Self {
id,
stats,
config,
tx: live_tx,
known_empty_since: None,
}
}
/// Mark the worker thread as idle from the present time if it reports no tasks.
///
/// This time information is used for thread culling.
#[inline]
pub fn try_mark_empty(&mut self, now: TokInstant) -> Option<TokInstant> {
if self.stats.live_mixers() == 0 {
self.known_empty_since.get_or_insert(now);
} else {
self.mark_busy();
}
self.known_empty_since
}
/// Unset the thread culling time on this worker.
#[inline]
pub fn mark_busy(&mut self) {
self.known_empty_since = None;
}
#[cfg(test)]
#[inline]
pub fn is_busy(&mut self) -> bool {
self.known_empty_since.is_none()
}
/// Return whether this thread has enough room (task count, spare cycles)
/// for the given task.
#[inline]
#[must_use]
pub fn can_schedule(&self, task: &ParkedMixer, avoid: Option<WorkerId>) -> bool {
avoid.is_none_or(|id| !self.has_id(id)) && self.stats.has_room(&self.config.strategy, task)
}
#[inline]
#[must_use]
pub fn stats(&self) -> Arc<LiveStatBlock> {
self.stats.clone()
}
/// Increment this worker's statistics and hand off a task for execution.
#[inline]
pub fn schedule_mixer(
&mut self,
id: TaskId,
task: ParkedMixer,
) -> Result<(), SendError<(TaskId, ParkedMixer)>> {
self.mark_busy();
self.stats.add_mixer();
self.tx.send((id, task))
}
#[must_use]
pub fn has_id(&self, id: WorkerId) -> bool {
self.id == id
}
}
const PACKETS_PER_BLOCK: usize = 16;
const MEMORY_CULL_TIMER: Duration = Duration::from_secs(10);
/// A synchronous thread responsible for mixing, encoding, encrypting, and
/// sending the audio output of many `Mixer`s.
///
/// `Mixer`s remain `Box`ed due to large move costs, and unboxing them appeared to have
/// a 5--10% perf cost from benchmarks.
pub struct Live {
packets: Vec<Box<[u8]>>,
packet_lens: Vec<usize>,
#[allow(clippy::vec_box)]
tasks: Vec<Box<Mixer>>,
ids: Vec<TaskId>,
to_cull: Vec<bool>,
deadline: Instant,
start_of_work: Option<Instant>,
id: WorkerId,
config: Config,
stats: Arc<LiveStatBlock>,
global_stats: Arc<StatBlock>,
rx: Receiver<(TaskId, ParkedMixer)>,
tx: Sender<SchedulerMessage>,
excess_buffer_cull_time: Option<Instant>,
}
#[allow(missing_docs)]
impl Live {
pub fn new(
id: WorkerId,
config: Config,
global_stats: Arc<StatBlock>,
stats: Arc<LiveStatBlock>,
rx: Receiver<(TaskId, ParkedMixer)>,
tx: Sender<SchedulerMessage>,
) -> Self {
let to_prealloc = config.strategy.prealloc_size();
let block_size = config
.strategy
.task_limit()
.unwrap_or(PACKETS_PER_BLOCK)
.min(PACKETS_PER_BLOCK);
let packets = vec![packet_block(block_size)];
Self {
packets,
packet_lens: Vec::with_capacity(to_prealloc),
tasks: Vec::with_capacity(to_prealloc),
ids: Vec::with_capacity(to_prealloc),
to_cull: Vec::with_capacity(to_prealloc),
deadline: Instant::now(),
start_of_work: None,
id,
config,
stats,
global_stats,
rx,
tx,
excess_buffer_cull_time: None,
}
}
#[inline]
fn run(&mut self) {
while self.run_once() {}
self.global_stats.remove_worker();
}
/// Returns whether the loop should exit (i.e., culled by main `Scheduler`).
#[inline]
pub fn run_once(&mut self) -> bool {
// Check for new tasks.
if self.handle_scheduler_msgs().is_err() {
return false;
}
// Receive commands for each task.
self.handle_task_msgs();
// Move any idle calls back to the global pool.
self.demote_and_remove_mixers();
// Take a clock measure before and after each packet.
let mut pre_pkt_time = Instant::now();
let mut worst_task = (0, Duration::default());
for (i, (packet_len, mixer)) in self
.packet_lens
.iter_mut()
.zip(self.tasks.iter_mut())
.enumerate()
{
let (block, inner) = get_memory_indices(i);
match mixer.mix_and_build_packet(&mut self.packets[block][inner..][..VOICE_PACKET_MAX])
{
Ok(written_sz) => *packet_len = written_sz,
e => {
*packet_len = 0;
rebuild_if_err(mixer, e, &mut self.to_cull, i);
},
}
let post_pkt_time = Instant::now();
let cost = post_pkt_time.duration_since(pre_pkt_time);
if cost > worst_task.1 {
worst_task = (i, cost);
}
pre_pkt_time = post_pkt_time;
}
let end_of_work = pre_pkt_time;
if let Some(start_of_work) = self.start_of_work {
let ns_cost = self.stats.store_compute_cost(end_of_work - start_of_work);
if self.config.move_expensive_tasks
&& ns_cost >= RESCHEDULE_THRESHOLD
&& self.ids.len() > 1
{
self.offload_mixer(worst_task.0, worst_task.1);
}
}
self.timed_remove_excess_blocks(end_of_work);
// Wait till the right time to send this packet:
// usually a 20ms tick, in test modes this is either a finite number of runs or user input.
self.march_deadline();
// Send all.
self.start_of_work = Some(Instant::now());
for (i, (packet_len, mixer)) in self
.packet_lens
.iter_mut()
.zip(self.tasks.iter_mut())
.enumerate()
{
let (block, inner) = get_memory_indices(i);
let packet = &mut self.packets[block][inner..];
if *packet_len > 0 {
let res = mixer.send_packet(&packet[..*packet_len]);
rebuild_if_err(mixer, res, &mut self.to_cull, i);
}
#[cfg(test)]
if *packet_len == 0 {
mixer.test_signal_empty_tick();
}
advance_rtp_counters(packet);
}
for (i, mixer) in self.tasks.iter_mut().enumerate() {
let res = mixer
.audio_commands_events()
.and_then(|()| mixer.check_and_send_keepalive(self.start_of_work));
rebuild_if_err(mixer, res, &mut self.to_cull, i);
}
true
}
#[cfg(test)]
fn _march_deadline(&mut self) {
// For testing, assume all will have same tick style.
// Only count 'remaining loops' on one of the nodes.
let mixer = self.tasks.get_mut(0).map(|m| {
let style = m.config.tick_style.clone();
(m, style)
});
match mixer {
None | Some((_, TickStyle::Timed)) => {
std::thread::sleep(self.deadline.saturating_duration_since(Instant::now()));
self.deadline += TIMESTEP_LENGTH;
},
Some((m, TickStyle::UntimedWithExecLimit(rx))) => {
if m.remaining_loops.is_none() {
if let Ok(new_val) = rx.recv() {
m.remaining_loops = Some(new_val.wrapping_sub(1));
}
}
if let Some(cnt) = m.remaining_loops.as_mut() {
if *cnt == 0 {
m.remaining_loops = None;
} else {
*cnt = cnt.wrapping_sub(1);
}
}
},
}
}
#[cfg(not(test))]
#[inline(always)]
#[allow(clippy::inline_always)]
fn _march_deadline(&mut self) {
std::thread::sleep(self.deadline.saturating_duration_since(Instant::now()));
self.deadline += TIMESTEP_LENGTH;
}
#[inline]
fn march_deadline(&mut self) {
#[cfg(feature = "internals")]
{
return;
}
#[allow(clippy::used_underscore_items)]
self._march_deadline();
}
#[inline]
fn handle_scheduler_msgs(&mut self) -> Result<(), ()> {
let mut activation_time = None;
loop {
match self.rx.try_recv() {
Ok((id, task)) => {
self.add_task(
task,
id,
*activation_time.get_or_insert_with(|| {
self.deadline
.checked_sub(TIMESTEP_LENGTH)
.unwrap_or(self.deadline)
}),
);
},
Err(TryRecvError::Empty) => break,
Err(TryRecvError::Disconnected) => return Err(()),
}
}
Ok(())
}
/// Handle messages from each tasks's `Driver`, marking dead tasks for removal.
#[inline]
fn handle_task_msgs(&mut self) {
for (i, (packet, mixer)) in self
.packets
.iter_mut()
.flat_map(|v| v.chunks_exact_mut(VOICE_PACKET_MAX))
.zip(self.tasks.iter_mut())
.enumerate()
{
let mut events_failure = false;
let mut conn_failure = false;
let fatal = loop {
match mixer.mix_rx.try_recv() {
Ok(m) => {
let (events, conn, should_exit) = mixer.handle_message(m, packet);
events_failure |= events;
conn_failure |= conn;
if should_exit {
break true;
}
},
Err(TryRecvError::Disconnected) => {
break true;
},
Err(TryRecvError::Empty) => {
break false;
},
}
};
if fatal || mixer.do_rebuilds(events_failure, conn_failure).is_err() {
// this is not zipped in because it is *not* needed most ticks.
self.to_cull[i] = true;
}
}
}
#[cfg(feature = "internals")]
#[inline]
pub fn mark_for_cull(&mut self, idx: usize) {
self.to_cull[idx] = true;
}
/// Check and demote for any tasks without live audio sources who have sent all
/// necessary silent frames (or remove dead tasks).
///
/// This must occur *after* handling per-track events to prevent erroneously
/// descheduling tasks.
#[inline]
pub fn demote_and_remove_mixers(&mut self) {
let mut i = 0;
while i < self.tasks.len() {
#[cfg(test)]
let force_conn = self.tasks[i].config.override_connection.is_some();
#[cfg(not(test))]
let force_conn = false;
// Benchmarking suggests that these asserts remove some bounds checks for us.
assert!(i < self.tasks.len());
assert!(i < self.to_cull.len());
if self.to_cull[i]
|| (self.tasks[i].tracks.is_empty() && self.tasks[i].silence_frames == 0)
|| !(self.tasks[i].conn_active.is_some() || force_conn)
{
self.stats.remove_mixer();
if let Some((id, parked)) = self.remove_task(i) {
self.global_stats.move_mixer_to_idle();
_ = self.tx.send(SchedulerMessage::Demote(id, parked));
} else {
self.global_stats.remove_live_mixer();
}
} else {
i += 1;
}
}
}
/// Return a given mixer to the main scheduler if this worker is overloaded.
#[inline]
pub fn offload_mixer(&mut self, idx: usize, cost: Duration) {
self.stats.remove_mixer();
if let Some((id, mut parked)) = self.remove_task(idx) {
self.global_stats.move_mixer_to_idle();
parked.last_cost = Some(cost);
_ = self
.tx
.send(SchedulerMessage::Overspill(self.id, id, parked));
} else {
self.global_stats.remove_live_mixer();
}
}
#[inline]
fn needed_blocks(&self) -> usize {
let div = self.ids.len() / PACKETS_PER_BLOCK;
let rem = self.ids.len() % PACKETS_PER_BLOCK;
(rem != 0) as usize + div
}
#[inline]
fn has_excess_blocks(&self) -> bool {
self.packets.len() > self.needed_blocks()
}
#[inline]
fn remove_excess_blocks(&mut self) {
self.packets.truncate(self.needed_blocks());
}
/// Try to offload excess packet buffers.
///
/// If there is currently overallocation, then store the first time at which
/// this was seenb. If this condition persists past `MEMORY_CULL_TIMER`, remove
/// unnecessary blocks.
#[inline]
fn timed_remove_excess_blocks(&mut self, now: Instant) {
if self.has_excess_blocks() {
if let Some(mark_time) = self.excess_buffer_cull_time {
if now.duration_since(mark_time) >= MEMORY_CULL_TIMER {
self.remove_excess_blocks();
self.excess_buffer_cull_time = None;
}
} else {
self.excess_buffer_cull_time = Some(now);
}
} else {
self.excess_buffer_cull_time = None;
}
}
#[inline]
fn add_task(&mut self, task: ParkedMixer, id: TaskId, activation_time: Instant) {
let idx = self.ids.len();
let elapsed = task.park_time - activation_time;
let samples_f64 = elapsed.as_secs_f64() * (SAMPLE_RATE_RAW as f64);
let mod_samples = (samples_f64 as u64) as u32;
let rtp_timestamp = task.rtp_timestamp.wrapping_add(mod_samples);
self.ids.push(id);
self.tasks.push(task.mixer);
self.packet_lens.push(0);
self.to_cull.push(false);
let (block, inner_idx) = get_memory_indices(idx);
while self.packets.len() <= block {
self.add_packet_block();
}
let packet = &mut self.packets[block][inner_idx..][..VOICE_PACKET_MAX];
let mut rtp = MutableRtpPacket::new(packet).expect(
"FATAL: Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
rtp.set_ssrc(task.ssrc);
rtp.set_timestamp(rtp_timestamp.into());
rtp.set_sequence(task.rtp_sequence.into());
}
/// Allocate and store a new packet block.
///
/// This will be full-size (`PACKETS_PER_BLOCK`) unless this block
/// is a) the last required for the task limit and b) this limit
/// is not aligned to `PACKETS_PER_BLOCK`.
#[inline]
fn add_packet_block(&mut self) {
let n_packets = if let Some(limit) = self.config.strategy.task_limit() {
let (block, inner) = get_memory_indices_unscaled(limit);
if self.packets.len() < block || inner == 0 {
PACKETS_PER_BLOCK
} else {
inner
}
} else {
PACKETS_PER_BLOCK
};
self.packets.push(packet_block(n_packets));
}
#[cfg(any(test, feature = "internals"))]
#[inline]
pub fn add_task_direct(&mut self, task: Mixer, id: TaskId) {
let id_0 = id.get();
self.add_task(
ParkedMixer {
mixer: Box::new(task),
ssrc: id_0 as u32,
rtp_sequence: id_0 as u16,
rtp_timestamp: id_0 as u32,
park_time: Instant::now(),
last_cost: None,
cull_handle: None,
},
id,
Instant::now(),
);
}
/// Remove a `Mixer`, returning it to the idle scheduler.
///
/// This operates by `swap_remove`ing each element of a Mixer's state, including
/// on RTP packet headers. This is achieved by setting up a memcpy between
/// buffer segments.
#[inline]
pub fn remove_task(&mut self, idx: usize) -> Option<(TaskId, ParkedMixer)> {
let end = self.tasks.len() - 1;
let id = self.ids.swap_remove(idx);
let _len = self.packet_lens.swap_remove(idx);
let mixer = self.tasks.swap_remove(idx);
let alive = !self.to_cull.swap_remove(idx);
let (block, inner_idx) = get_memory_indices(idx);
let (removed, replacement) = if end > idx {
let (end_block, end_inner) = get_memory_indices(end);
let (rest, target_block) = self.packets.split_at_mut(end_block);
let (last_block, end_pkt) = target_block[0].split_at_mut(end_inner);
if end_block == block {
(&mut last_block[inner_idx..], Some(end_pkt))
} else {
(&mut rest[block][inner_idx..], Some(end_pkt))
}
} else {
(&mut self.packets[block][inner_idx..], None)
};
let rtp = RtpPacket::new(removed).expect(
"FATAL: Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
let ssrc = rtp.get_ssrc();
let rtp_timestamp = rtp.get_timestamp().into();
let rtp_sequence = rtp.get_sequence().into();
if let Some(replacement) = replacement {
// Copy the whole packet header since we know it'll be 4B aligned.
// 'Just necessary fields' is 2B aligned.
const COPY_LEN: usize = RtpPacket::minimum_packet_size();
removed[..COPY_LEN].copy_from_slice(&replacement[..COPY_LEN]);
}
alive.then(move || {
let park_time = Instant::now();
(
id,
ParkedMixer {
mixer,
ssrc,
rtp_sequence,
rtp_timestamp,
park_time,
last_cost: None,
cull_handle: None,
},
)
})
}
/// Spawn a new sync thread to manage `Mixer`s.
fn spawn(mut self) {
std::thread::spawn(move || {
self.run();
});
}
}
/// Initialises a packet block of the required size, prefilling any constant RTP data.
#[inline]
fn packet_block(n_packets: usize) -> Box<[u8]> {
let mut packets = vec![0u8; VOICE_PACKET_MAX * n_packets].into_boxed_slice();
for packet in packets.chunks_exact_mut(VOICE_PACKET_MAX) {
let mut rtp = MutableRtpPacket::new(packet).expect(
"FATAL: Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
rtp.set_version(RTP_VERSION);
rtp.set_payload_type(RTP_PROFILE_TYPE);
}
packets
}
/// Returns the block index into `self.packets` and the packet number in
/// the block for a given worker's index.
#[inline]
fn get_memory_indices_unscaled(idx: usize) -> (usize, usize) {
let block_size = PACKETS_PER_BLOCK;
(idx / block_size, idx % block_size)
}
/// Returns the block index into `self.packets` and the byte offset into
/// a packet block for a given worker's index.
#[inline]
fn get_memory_indices(idx: usize) -> (usize, usize) {
let (block, inner_unscaled) = get_memory_indices_unscaled(idx);
(block, inner_unscaled * VOICE_PACKET_MAX)
}
#[inline]
fn advance_rtp_counters(packet: &mut [u8]) {
let mut rtp = MutableRtpPacket::new(packet).expect(
"FATAL: Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
rtp.set_sequence(rtp.get_sequence() + 1);
rtp.set_timestamp(rtp.get_timestamp() + MONO_FRAME_SIZE as u32);
}
/// Structured slightly confusingly: we only want to even access `cull_markers`
/// in the event of error.
#[inline]
fn rebuild_if_err<T>(
mixer: &mut Box<Mixer>,
res: Result<T, DriverError>,
cull_markers: &mut [bool],
idx: usize,
) {
if let Err(e) = res {
cull_markers[idx] |= mixer
.do_rebuilds(
e.should_trigger_interconnect_rebuild(),
e.should_trigger_connect(),
)
.is_err();
}
}
#[cfg(test)]
mod test {
use super::*;
use crate::driver::test_impls::*;
use tokio::runtime::Handle;
fn rtp_has_index(pkt: &[u8], sentinel_val: u16) {
let rtp = RtpPacket::new(pkt).unwrap();
assert_eq!(rtp.get_version(), RTP_VERSION);
assert_eq!(rtp.get_padding(), 0);
assert_eq!(rtp.get_extension(), 0);
assert_eq!(rtp.get_csrc_count(), 0);
assert_eq!(rtp.get_marker(), 0);
assert_eq!(rtp.get_payload_type(), RTP_PROFILE_TYPE);
assert_eq!(rtp.get_sequence(), sentinel_val.into());
assert_eq!(rtp.get_timestamp(), (sentinel_val as u32).into());
assert_eq!(rtp.get_ssrc(), sentinel_val as u32);
}
#[tokio::test]
async fn block_alloc_is_partial_small() {
let n_mixers = 1;
let (sched, _listeners) = MockScheduler::from_mixers(
Some(Mode::MaxPerThread(n_mixers.try_into().unwrap())),
(0..n_mixers)
.map(|_| Mixer::test_with_float(1, Handle::current(), false))
.collect(),
);
assert_eq!(sched.core.packets.len(), 1);
assert_eq!(sched.core.packets[0].len(), VOICE_PACKET_MAX);
}
#[tokio::test]
async fn block_alloc_is_partial_large() {
let n_mixers = 33;
let (sched, _listeners) = MockScheduler::from_mixers(
Some(Mode::MaxPerThread(n_mixers.try_into().unwrap())),
(0..n_mixers)
.map(|_| Mixer::test_with_float(1, Handle::current(), false))
.collect(),
);
assert_eq!(sched.core.packets.len(), 3);
assert_eq!(
sched.core.packets[0].len(),
PACKETS_PER_BLOCK * VOICE_PACKET_MAX
);
assert_eq!(
sched.core.packets[1].len(),
PACKETS_PER_BLOCK * VOICE_PACKET_MAX
);
assert_eq!(sched.core.packets[2].len(), VOICE_PACKET_MAX);
}
#[tokio::test]
async fn deletion_moves_pkt_header() {
let (mut sched, _listeners) = MockScheduler::from_mixers(
None,
(0..PACKETS_PER_BLOCK)
.map(|_| Mixer::test_with_float(1, Handle::current(), false))
.collect(),
);
let last_idx = (PACKETS_PER_BLOCK - 1) as u16;
// Remove head.
sched.core.remove_task(0);
rtp_has_index(&sched.core.packets[0], last_idx);
// Remove head.
sched.core.remove_task(5);
rtp_has_index(&sched.core.packets[0][5 * VOICE_PACKET_MAX..], last_idx - 1);
}
#[tokio::test]
async fn deletion_moves_pkt_header_multiblock() {
let n_pkts = PACKETS_PER_BLOCK + 8;
let (mut sched, _listeners) = MockScheduler::from_mixers(
None,
(0..n_pkts)
.map(|_| Mixer::test_with_float(1, Handle::current(), false))
.collect(),
);
let last_idx = (n_pkts - 1) as u16;
// Remove head (read from block 1 into block 0).
sched.core.remove_task(0);
rtp_has_index(&sched.core.packets[0], last_idx);
// Remove later (read from block 1 into block 1).
sched.core.remove_task(17);
rtp_has_index(&sched.core.packets[1][VOICE_PACKET_MAX..], last_idx - 1);
}
#[tokio::test]
async fn packet_blocks_are_cleaned_up() {
// Allocate 2 blocks.
let n_pkts = PACKETS_PER_BLOCK + 1;
let (mut sched, _listeners) = MockScheduler::from_mixers(
None,
(0..n_pkts)
.map(|_| Mixer::test_with_float(1, Handle::current(), false))
.collect(),
);
// Assert no cleanup at start.
assert!(sched.core.run_once());
assert_eq!(sched.core.needed_blocks(), 2);
assert!(sched.core.excess_buffer_cull_time.is_none());
// Remove only entry in last block. Cleanup should be sched'd.
sched.core.remove_task(n_pkts - 1);
assert!(sched.core.run_once());
assert!(sched.core.has_excess_blocks());
assert!(sched.core.excess_buffer_cull_time.is_some());
tokio::time::sleep(Duration::from_secs(2) + MEMORY_CULL_TIMER).await;
// Cleanup should be unsched'd.
assert!(sched.core.run_once());
assert!(sched.core.excess_buffer_cull_time.is_none());
assert!(!sched.core.has_excess_blocks());
}
}

View File

@@ -0,0 +1,180 @@
use std::{
error::Error as StdError,
fmt::Display,
num::NonZeroUsize,
sync::{Arc, OnceLock},
};
use flume::{Receiver, RecvError, Sender};
use super::tasks::message::{Interconnect, MixerMessage};
use crate::{constants::TIMESTEP_LENGTH, Config as DriverConfig};
mod config;
mod idle;
mod live;
mod stats;
mod task;
pub use config::*;
use idle::*;
pub use live::*;
pub use stats::*;
pub use task::*;
/// A soft maximum of 90% of the 20ms budget to account for variance in execution time.
const RESCHEDULE_THRESHOLD: u64 = ((TIMESTEP_LENGTH.subsec_nanos() as u64) * 9) / 10;
const DEFAULT_MIXERS_PER_THREAD: NonZeroUsize = match NonZeroUsize::new(16) {
Some(v) => v,
None => unreachable!(),
};
/// The default shared scheduler instance.
///
/// This is built using the default value of [`ScheduleMode`]. Users desiring
/// a custom strategy should avoid calling [`Config::default`].
///
/// [`Config::default`]: crate::Config::default
/// [`ScheduleMode`]: Mode
pub fn get_default_scheduler() -> &'static Scheduler {
static DEFAULT_SCHEDULER: OnceLock<Scheduler> = OnceLock::new();
DEFAULT_SCHEDULER.get_or_init(Scheduler::default)
}
/// A reference to a shared group of threads used for running idle and active
/// audio threads.
#[derive(Clone, Debug)]
pub struct Scheduler {
inner: Arc<InnerScheduler>,
}
/// Inner contents of a [`Scheduler`] instance.
///
/// This is an `Arc` around `Arc`'d contents so that we can make use of the
/// drop check on `Scheduler` to clean up resources.
#[derive(Clone, Debug)]
struct InnerScheduler {
tx: Sender<SchedulerMessage>,
stats: Arc<StatBlock>,
}
impl Scheduler {
/// Create a new mixer scheduler from the allocation strategy in `config`.
#[must_use]
pub fn new(config: Config) -> Self {
let (core, tx) = Idle::new(config);
let stats = core.stats.clone();
core.spawn();
let inner = Arc::new(InnerScheduler { tx, stats });
Self { inner }
}
pub(crate) fn new_mixer(
&self,
config: &DriverConfig,
ic: Interconnect,
rx: Receiver<MixerMessage>,
) {
self.inner
.tx
.send(SchedulerMessage::NewMixer(rx, ic, config.clone()))
.unwrap();
}
/// Returns the total number of calls (idle and active) scheduled.
#[must_use]
pub fn total_tasks(&self) -> u64 {
self.inner.stats.total_mixers()
}
/// Returns the total number of *active* calls scheduled and processing
/// audio.
#[must_use]
pub fn live_tasks(&self) -> u64 {
self.inner.stats.live_mixers()
}
/// Returns the total number of threads spawned to process live audio sessions.
#[must_use]
pub fn worker_threads(&self) -> u64 {
self.inner.stats.worker_threads()
}
/// Request a list of handles to statistics for currently live workers.
pub async fn worker_thread_stats(&self) -> Result<Vec<Arc<LiveStatBlock>>, Error> {
let (tx, rx) = flume::bounded(1);
_ = self.inner.tx.send(SchedulerMessage::GetStats(tx));
rx.recv_async().await.map_err(Error::from)
}
/// Request a list of handles to statistics for currently live workers with a blocking call.
pub fn worker_thread_stats_blocking(&self) -> Result<Vec<Arc<LiveStatBlock>>, Error> {
let (tx, rx) = flume::bounded(1);
_ = self.inner.tx.send(SchedulerMessage::GetStats(tx));
rx.recv().map_err(Error::from)
}
}
impl Drop for InnerScheduler {
fn drop(&mut self) {
_ = self.tx.send(SchedulerMessage::Kill);
}
}
impl Default for Scheduler {
fn default() -> Self {
Scheduler::new(Config::default())
}
}
/// Control messages for a scheduler.
pub enum SchedulerMessage {
/// Build a new `Mixer` as part of the initialisation of a `Driver`.
NewMixer(Receiver<MixerMessage>, Interconnect, DriverConfig),
/// Forward a command for
Do(TaskId, MixerMessage),
/// Return a `Mixer` from a worker back to the idle pool.
Demote(TaskId, ParkedMixer),
/// Move an expensive `Mixer` to another thread in the worker pool.
Overspill(WorkerId, TaskId, ParkedMixer),
/// Request a copy of all per-worker statistics.
GetStats(Sender<Vec<Arc<LiveStatBlock>>>),
/// Cleanup once all `Scheduler` handles are dropped.
Kill,
}
/// Errors encountered when communicating with the internals of a [`Scheduler`].
///
/// [`Scheduler`]: crate::driver::Scheduler
#[non_exhaustive]
#[derive(Debug)]
pub enum Error {
/// The scheduler exited or crashed while awating the request.
Disconnected,
}
impl Display for Error {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Disconnected => f.write_str("the scheduler terminated mid-request"),
}
}
}
impl StdError for Error {
fn source(&self) -> Option<&(dyn StdError + 'static)> {
None
}
}
impl From<RecvError> for Error {
fn from(_: RecvError) -> Self {
Self::Disconnected
}
}

View File

@@ -0,0 +1,134 @@
use std::{
sync::atomic::{AtomicU64, Ordering},
time::Duration,
};
use super::{Mode, ParkedMixer, RESCHEDULE_THRESHOLD};
/// Statistics shared by an entire `Scheduler`.
#[derive(Debug, Default)]
pub struct StatBlock {
total: AtomicU64,
live: AtomicU64,
threads: AtomicU64,
}
#[allow(missing_docs)]
impl StatBlock {
#[inline]
pub fn total_mixers(&self) -> u64 {
self.total.load(Ordering::Relaxed)
}
#[inline]
pub fn live_mixers(&self) -> u64 {
self.live.load(Ordering::Relaxed)
}
#[inline]
pub fn worker_threads(&self) -> u64 {
self.threads.load(Ordering::Relaxed)
}
#[inline]
pub fn add_idle_mixer(&self) {
self.total.fetch_add(1, Ordering::Relaxed);
}
#[inline]
pub fn remove_idle_mixers(&self, n: u64) {
self.total.fetch_sub(n, Ordering::Relaxed);
}
#[inline]
pub fn move_mixer_to_live(&self) {
self.live.fetch_add(1, Ordering::Relaxed);
}
#[inline]
pub fn move_mixer_to_idle(&self) {
self.move_mixers_to_idle(1);
}
#[inline]
pub fn move_mixers_to_idle(&self, n: u64) {
self.live.fetch_sub(n, Ordering::Relaxed);
}
#[inline]
pub fn remove_live_mixer(&self) {
self.remove_live_mixers(1);
}
#[inline]
pub fn remove_live_mixers(&self, n: u64) {
self.move_mixers_to_idle(n);
self.remove_idle_mixers(n);
}
#[inline]
pub fn add_worker(&self) {
self.threads.fetch_add(1, Ordering::Relaxed);
}
#[inline]
pub fn remove_worker(&self) {
self.threads.fetch_sub(1, Ordering::Relaxed);
}
}
/// Runtime statistics for an individual worker.
///
/// Individual statistics are measured atomically -- the worker thread
/// may have been cleaned up, or its mixer count may not match the
/// count when [`Self::last_compute_cost_ns`] was set.
#[derive(Debug, Default)]
pub struct LiveStatBlock {
live: AtomicU64,
last_ns: AtomicU64,
}
impl LiveStatBlock {
/// Returns the number of mixer tasks scheduled on this worker thread.
#[inline]
pub fn live_mixers(&self) -> u64 {
self.live.load(Ordering::Relaxed)
}
#[inline]
pub(crate) fn add_mixer(&self) {
self.live.fetch_add(1, Ordering::Relaxed);
}
#[inline]
pub(crate) fn remove_mixer(&self) {
self.live.fetch_sub(1, Ordering::Relaxed);
}
#[inline]
pub(crate) fn store_compute_cost(&self, work: Duration) -> u64 {
let cost = work.as_nanos() as u64;
self.last_ns.store(cost, Ordering::Relaxed);
cost
}
/// Returns the number of nanoseconds required to process all worker threads'
/// packet transmission, mixing, encoding, and encryption in the last tick.
#[inline]
pub fn last_compute_cost_ns(&self) -> u64 {
self.last_ns.load(Ordering::Relaxed)
}
#[inline]
pub(crate) fn has_room(&self, strategy: &Mode, task: &ParkedMixer) -> bool {
let task_room = strategy
.task_limit()
.is_none_or(|limit| self.live_mixers() < limit as u64);
let exec_room = task.last_cost.is_none_or(|cost| {
cost.as_nanos() as u64 + self.last_compute_cost_ns() < RESCHEDULE_THRESHOLD
});
task_room && exec_room
}
}

View File

@@ -0,0 +1,214 @@
use std::{
marker::PhantomData,
time::{Duration, Instant},
};
use flume::{Receiver, Sender};
use nohash_hasher::IsEnabled;
use rand::random;
use tokio::runtime::Handle;
use crate::{
driver::tasks::{
error::Error as DriverError,
message::{EventMessage, Interconnect, MixerMessage},
mixer::Mixer,
},
Config,
};
use super::SchedulerMessage;
/// Typesafe counter used to identify individual mixer/worker instances.
#[derive(Copy, Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct ResId<T>(u64, PhantomData<T>);
#[allow(missing_docs)]
pub type TaskId = ResId<TaskMarker>;
#[allow(missing_docs)]
pub type WorkerId = ResId<WorkerMarker>;
#[allow(missing_docs)]
#[derive(Copy, Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct TaskMarker;
#[allow(missing_docs)]
#[derive(Copy, Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct WorkerMarker;
impl<T> IsEnabled for ResId<T> {}
#[allow(missing_docs)]
impl<T: Copy> ResId<T> {
#[must_use]
pub fn new() -> Self {
Self::default()
}
pub fn incr(&mut self) -> Self {
let out = *self;
self.0 = self.0.wrapping_add(1);
out
}
#[cfg(any(test, feature = "internals"))]
#[must_use]
pub fn get(self) -> u64 {
self.0
}
}
impl<T: Copy> Default for ResId<T> {
fn default() -> Self {
Self(0, PhantomData)
}
}
/// An idle mixer instance, externally controlled by a `Driver`.
///
/// Since we do not allocate packet buffers for idle threads, this
/// struct includes various RTP fields.
pub struct ParkedMixer {
/// Mixer, track, etc. state as well as message receivers.
pub mixer: Box<Mixer>,
/// The SSRC assigned to this voice session.
pub ssrc: u32,
/// The last recorded/generated RTP sequence.
pub rtp_sequence: u16,
/// The last recorded/generated RTP timestamp.
pub rtp_timestamp: u32,
/// The time at which this `Mixer` was made idle.
///
/// This is used when transitioning to a live state to determine
/// how far we should adjust the RTP timestamp by.
pub park_time: Instant,
/// The last known cost of executing this task, if it had to be moved
/// due to a limit on thread resources.
pub last_cost: Option<Duration>,
/// Handle to any forwarder task, used if this mixer is culled while idle.
pub cull_handle: Option<Sender<()>>,
}
#[allow(missing_docs)]
impl ParkedMixer {
/// Create a new `Mixer` in a parked state.
#[must_use]
pub fn new(mix_rx: Receiver<MixerMessage>, interconnect: Interconnect, config: Config) -> Self {
Self {
mixer: Box::new(Mixer::new(mix_rx, Handle::current(), interconnect, config)),
ssrc: 0,
rtp_sequence: random::<u16>(),
rtp_timestamp: random::<u32>(),
park_time: Instant::now(),
last_cost: None,
cull_handle: None,
}
}
/// Spawn a tokio task which forwards any mixer messages to the central `Idle` task pool.
///
/// Any requests which would cause this mixer to become live will terminate
/// this task.
pub fn spawn_forwarder(&mut self, tx: Sender<SchedulerMessage>, id: TaskId) {
let (kill_tx, kill_rx) = flume::bounded(1);
self.cull_handle = Some(kill_tx);
let remote_rx = self.mixer.mix_rx.clone();
tokio::spawn(async move {
loop {
tokio::select! {
biased;
_ = kill_rx.recv_async() => break,
msg = remote_rx.recv_async() => {
let exit = if let Ok(msg) = msg {
let remove_self = msg.is_mixer_maybe_live();
tx.send_async(SchedulerMessage::Do(id, msg)).await.is_err() || remove_self
} else {
true
};
if exit {
break;
}
}
}
}
});
}
/// Returns whether the mixer should exit and be cleaned up.
pub fn handle_message(&mut self, msg: MixerMessage) -> Result<bool, ()> {
match msg {
MixerMessage::SetConn(conn, ssrc) => {
// Overridden because payload-specific fields are carried
// externally on `ParkedMixer`.
self.ssrc = ssrc;
self.rtp_sequence = random::<u16>();
self.rtp_timestamp = random::<u32>();
self.mixer.conn_active = Some(conn);
self.mixer.update_keepalive(ssrc);
Ok(false)
},
MixerMessage::Ws(ws) => {
// Overridden so that we don't mistakenly tell Discord we're speaking.
self.mixer.ws = ws;
self.send_gateway_not_speaking();
Ok(false)
},
msg => {
let (events_failure, conn_failure, should_exit) =
self.mixer.handle_message(msg, &mut []);
self.mixer
.do_rebuilds(events_failure, conn_failure)
.map_err(|_| ())
.map(|()| should_exit)
},
}
}
/// Handle periodic events attached to this `Mixer`, including timer state
/// on the event thread and UDP keepalives needed to prevent session termination.
///
/// As we init our UDP sockets as non-blocking via Tokio -> `into_std`, it is
/// safe to call UDP packet sends like this.
pub fn tick_and_keepalive(&mut self, now: Instant) -> Result<(), ()> {
// TODO: should we include an atomic which signals whether the event
// thread *cares*, so we can prevent wakeups?
// Can we do the same for live tracks?
let mut events_failure = self.mixer.fire_event(EventMessage::Tick).is_err();
let ka_err = self
.mixer
.check_and_send_keepalive(Some(now))
.or_else(DriverError::disarm_would_block);
let conn_failure = if let Err(e) = ka_err {
events_failure |= e.should_trigger_interconnect_rebuild();
e.should_trigger_connect()
} else {
false
};
self.mixer
.do_rebuilds(events_failure, conn_failure)
.map_err(|_| ())
}
pub fn send_gateway_speaking(&mut self) -> Result<(), ()> {
if let Err(e) = self.mixer.send_gateway_speaking() {
self.mixer
.do_rebuilds(
e.should_trigger_interconnect_rebuild(),
e.should_trigger_connect(),
)
.map_err(|_| ())
} else {
Ok(())
}
}
pub fn send_gateway_not_speaking(&self) {
self.mixer.send_gateway_not_speaking();
}
}

View File

@@ -0,0 +1,44 @@
use super::message::*;
use flume::{Receiver, Sender};
use tracing::{instrument, trace};
#[derive(Debug, Clone)]
pub struct DisposalThread(Sender<DisposalMessage>);
impl Default for DisposalThread {
fn default() -> Self {
Self::run()
}
}
impl DisposalThread {
#[must_use]
pub fn run() -> Self {
let (mix_tx, mix_rx) = flume::unbounded();
std::thread::spawn(move || {
trace!("Disposal thread started.");
runner(mix_rx);
trace!("Disposal thread finished.");
});
Self(mix_tx)
}
pub(super) fn dispose(&self, message: DisposalMessage) {
drop(self.0.send(message));
}
}
/// The mixer's disposal thread is also synchronous, due to tracks,
/// inputs, etc. being based on synchronous I/O.
///
/// The mixer uses this to offload heavy and expensive drop operations
/// to prevent deadline misses.
#[instrument(skip(mix_rx))]
#[expect(
clippy::needless_pass_by_value,
reason = "spawned on background thread, must take by value"
)]
fn runner(mix_rx: Receiver<DisposalMessage>) {
while mix_rx.recv().is_ok() {}
}

View File

@@ -0,0 +1,150 @@
use super::message::*;
use crate::ws::Error as WsError;
use aes_gcm::Error as CryptoError;
use davey::errors::EncryptError as DaveyEncryptError;
use flume::SendError;
use opus2::Error as OpusError;
use std::io::{Error as IoError, ErrorKind as IoErrorKind};
#[derive(Debug)]
pub enum Recipient {
AuxNetwork,
Event,
Mixer,
#[cfg(feature = "receive")]
UdpRx,
}
pub type Result<T> = std::result::Result<T, Error>;
#[derive(Debug)]
#[non_exhaustive]
pub enum Error {
Crypto(CryptoError),
DaveEncrypt,
#[cfg(any(feature = "receive", test))]
/// Received an illegal voice packet on the voice UDP socket.
IllegalVoicePacket,
InterconnectFailure(Recipient),
Io(IoError),
Other,
}
impl Error {
pub(crate) fn should_trigger_connect(&self) -> bool {
match self {
Error::InterconnectFailure(Recipient::AuxNetwork) => true,
#[cfg(feature = "receive")]
Error::InterconnectFailure(Recipient::UdpRx) => true,
_ => false,
}
}
pub(crate) fn should_trigger_interconnect_rebuild(&self) -> bool {
matches!(self, Error::InterconnectFailure(Recipient::Event))
}
// This prevents a `WouldBlock` from triggering a full reconnect,
// instead simply dropping the packet.
pub(crate) fn disarm_would_block(self) -> Result<()> {
match self {
Self::Io(i) if i.kind() == IoErrorKind::WouldBlock => Ok(()),
e => Err(e),
}
}
}
impl From<CryptoError> for Error {
fn from(e: CryptoError) -> Self {
Error::Crypto(e)
}
}
impl From<DaveyEncryptError> for Error {
fn from(_e: DaveyEncryptError) -> Self {
Self::DaveEncrypt
}
}
impl From<IoError> for Error {
fn from(e: IoError) -> Error {
Error::Io(e)
}
}
impl From<OpusError> for Error {
fn from(_: OpusError) -> Error {
Error::Other
}
}
impl From<SendError<WsMessage>> for Error {
fn from(_e: SendError<WsMessage>) -> Error {
Error::InterconnectFailure(Recipient::AuxNetwork)
}
}
impl From<SendError<EventMessage>> for Error {
fn from(_e: SendError<EventMessage>) -> Error {
Error::InterconnectFailure(Recipient::Event)
}
}
impl From<SendError<MixerMessage>> for Error {
fn from(_e: SendError<MixerMessage>) -> Error {
Error::InterconnectFailure(Recipient::Mixer)
}
}
#[cfg(feature = "receive")]
impl From<SendError<UdpRxMessage>> for Error {
fn from(_e: SendError<UdpRxMessage>) -> Error {
Error::InterconnectFailure(Recipient::UdpRx)
}
}
impl From<WsError> for Error {
fn from(_: WsError) -> Error {
Error::Other
}
}
#[derive(Debug)]
#[non_exhaustive]
pub enum DaveReinitError {
Init,
Reinit,
Reset,
CreateKeyPackage,
Ws(WsError),
}
impl From<davey::errors::InitError> for DaveReinitError {
fn from(_e: davey::errors::InitError) -> Self {
Self::Init
}
}
impl From<davey::errors::ReinitError> for DaveReinitError {
fn from(_e: davey::errors::ReinitError) -> Self {
Self::Reinit
}
}
impl From<davey::errors::ResetError> for DaveReinitError {
fn from(_e: davey::errors::ResetError) -> Self {
Self::Reset
}
}
impl From<davey::errors::CreateKeyPackageError> for DaveReinitError {
fn from(_e: davey::errors::CreateKeyPackageError) -> Self {
Self::CreateKeyPackage
}
}
impl From<WsError> for DaveReinitError {
fn from(e: WsError) -> Self {
Self::Ws(e)
}
}

View File

@@ -0,0 +1,139 @@
use super::message::*;
use crate::{
events::{EventStore, GlobalEvents, TrackEvent},
tracks::{ReadyState, TrackHandle, TrackState},
};
use flume::Receiver;
use tracing::{debug, info, instrument, trace};
#[instrument(skip(evt_rx))]
pub(crate) async fn runner(evt_rx: Receiver<EventMessage>) {
let mut global = GlobalEvents::default();
let mut events: Vec<EventStore> = vec![];
let mut states: Vec<TrackState> = vec![];
let mut handles: Vec<TrackHandle> = vec![];
while let Ok(msg) = evt_rx.recv_async().await {
match msg {
EventMessage::AddGlobalEvent(data) => {
info!("Global event added.");
global.add_event(data);
},
EventMessage::AddTrackEvent(i, data) => {
info!("Adding event to track {}.", i);
let event_store = events
.get_mut(i)
.expect("Event thread was given an illegal store index for AddTrackEvent.");
let state = states
.get_mut(i)
.expect("Event thread was given an illegal state index for AddTrackEvent.");
event_store.add_event(data, state.position);
},
EventMessage::FireCoreEvent(ctx) => {
let ctx = ctx.to_user_context();
let evt = ctx
.to_core_event()
.expect("Event thread was passed a non-core event in FireCoreEvent.");
trace!("Firing core event {:?}.", evt);
global.fire_core_event(evt, ctx).await;
},
EventMessage::RemoveGlobalEvents => {
global.remove_handlers();
},
EventMessage::AddTrack(store, state, handle) => {
events.push(store);
states.push(state);
handles.push(handle);
info!("Event state for track {} added", events.len());
},
EventMessage::ChangeState(i, change) => {
let max_states = states.len();
debug!(
"Changing state for track {} of {}: {:?}",
i, max_states, change
);
let state = states
.get_mut(i)
.expect("Event thread was given an illegal state index for ChangeState.");
match change {
TrackStateChange::Mode(mut mode) => {
std::mem::swap(&mut state.playing, &mut mode);
if state.playing != mode {
global.fire_track_event(state.playing.as_track_event(), i);
if let Some(other_evts) = state.playing.also_fired_track_events() {
for evt in other_evts {
global.fire_track_event(evt, i);
}
}
}
},
TrackStateChange::Volume(vol) => {
state.volume = vol;
},
TrackStateChange::Position(pos) => {
// Currently, only Tick should fire time events.
state.position = pos;
},
TrackStateChange::Loops(loops, user_set) => {
state.loops = loops;
if !user_set {
global.fire_track_event(TrackEvent::Loop, i);
}
},
TrackStateChange::Total(new) => {
// Massive, unprecedented state changes.
*state = new;
},
TrackStateChange::Ready(ready_state) => {
state.ready = ready_state;
match ready_state {
ReadyState::Playable => {
global.fire_track_event(TrackEvent::Playable, i);
},
ReadyState::Preparing => {
global.fire_track_event(TrackEvent::Preparing, i);
},
ReadyState::Uninitialised => {},
}
},
}
},
EventMessage::RemoveAllTracks => {
info!("Event state for all tracks removed.");
events.clear();
states.clear();
handles.clear();
},
EventMessage::Tick => {
// NOTE: this should fire saved up blocks of state change evts.
global.tick(&mut events, &mut states, &mut handles).await;
let mut i = 0;
while i < states.len() {
if states[i].playing.is_done() {
info!("Event state for track {} of {} removed.", i, events.len());
events.swap_remove(i);
states.swap_remove(i);
handles.swap_remove(i);
} else {
i += 1;
}
}
},
EventMessage::Poison => break,
}
}
trace!("Event thread exited.");
}

View File

@@ -0,0 +1,33 @@
#![allow(missing_docs)]
use crate::{
driver::{connection::error::Error, Bitrate, Config},
events::{context_data::DisconnectReason, EventData},
tracks::{Track, TrackCommand, TrackHandle},
ConnectionInfo,
};
use flume::{Receiver, Sender};
pub enum CoreMessage {
ConnectWithResult(ConnectionInfo, Sender<Result<(), Error>>),
RetryConnect(usize),
SignalWsClosure(usize, ConnectionInfo, Option<DisconnectReason>),
Disconnect,
SetTrack(Option<Box<TrackContext>>),
AddTrack(Box<TrackContext>),
SetBitrate(Bitrate),
AddEvent(EventData),
RemoveGlobalEvents,
SetConfig(Config),
Mute(bool),
Reconnect,
FullReconnect,
RebuildInterconnect,
Poison,
}
pub struct TrackContext {
pub track: Track,
pub handle: TrackHandle,
pub receiver: Receiver<TrackCommand>,
}

View File

@@ -0,0 +1,9 @@
#![allow(missing_docs)]
use crate::{driver::tasks::mixer::InternalTrack, tracks::TrackHandle};
#[allow(dead_code)] // We don't read because all we are doing is dropping.
pub enum DisposalMessage {
Track(Box<InternalTrack>),
Handle(TrackHandle),
}

View File

@@ -0,0 +1,34 @@
#![allow(missing_docs)]
use crate::{
events::{CoreContext, EventData, EventStore},
tracks::{LoopState, PlayMode, ReadyState, TrackHandle, TrackState},
};
use std::time::Duration;
pub enum EventMessage {
// Event related.
// Track events should fire off the back of state changes.
AddGlobalEvent(EventData),
AddTrackEvent(usize, EventData),
FireCoreEvent(CoreContext),
RemoveGlobalEvents,
AddTrack(EventStore, TrackState, TrackHandle),
ChangeState(usize, TrackStateChange),
RemoveAllTracks,
Tick,
Poison,
}
#[derive(Debug)]
pub enum TrackStateChange {
Mode(PlayMode),
Volume(f32),
Position(Duration),
// Bool indicates user-set.
Loops(LoopState, bool),
Total(TrackState),
Ready(ReadyState),
}

View File

@@ -0,0 +1,65 @@
#![allow(missing_docs)]
#[cfg(feature = "receive")]
use super::UdpRxMessage;
use super::{Interconnect, TrackContext, WsMessage};
use crate::{
driver::{crypto::Cipher, Bitrate, Config, CryptoState},
input::{AudioStreamError, Compose, Parsed},
};
use flume::Sender;
use std::{
net::UdpSocket,
sync::{atomic::AtomicU16, Arc, RwLock},
};
use symphonia_core::{errors::Error as SymphoniaError, formats::SeekedTo};
pub struct MixerConnection {
pub cipher: Cipher,
pub crypto_state: CryptoState,
pub dave_session: Arc<RwLock<Option<davey::DaveSession>>>,
pub dave_protocol_version: Arc<AtomicU16>,
#[cfg(feature = "receive")]
pub udp_rx: Sender<UdpRxMessage>,
pub udp_tx: UdpSocket,
}
pub enum MixerMessage {
AddTrack(Box<TrackContext>),
SetTrack(Option<Box<TrackContext>>),
SetBitrate(Bitrate),
SetConfig(Config),
SetMute(bool),
SetConn(MixerConnection, u32),
Ws(Option<Sender<WsMessage>>),
DropConn,
ReplaceInterconnect(Interconnect),
RebuildEncoder,
Poison,
}
impl MixerMessage {
#[must_use]
pub fn is_mixer_maybe_live(&self) -> bool {
matches!(
self,
Self::AddTrack(_) | Self::SetTrack(Some(_)) | Self::SetConn(..)
)
}
}
pub enum MixerInputResultMessage {
CreateErr(Arc<AudioStreamError>),
ParseErr(Arc<SymphoniaError>),
Seek(
Parsed,
Option<Box<dyn Compose>>,
Result<SeekedTo, Arc<SymphoniaError>>,
),
Built(Parsed, Option<Box<dyn Compose>>),
}

View File

@@ -0,0 +1,55 @@
#![allow(missing_docs)]
mod core;
mod disposal;
mod events;
mod mixer;
#[cfg(feature = "receive")]
mod udp_rx;
mod ws;
#[cfg(feature = "receive")]
pub use self::udp_rx::*;
pub use self::{core::*, disposal::*, events::*, mixer::*, ws::*};
use flume::Sender;
use tokio::spawn;
use tracing::trace;
#[derive(Clone, Debug)]
pub struct Interconnect {
pub core: Sender<CoreMessage>,
pub events: Sender<EventMessage>,
pub mixer: Sender<MixerMessage>,
}
impl Interconnect {
pub fn poison(&self) {
drop(self.events.send(EventMessage::Poison));
}
pub fn poison_all(&self) {
drop(self.mixer.send(MixerMessage::Poison));
self.poison();
}
pub fn restart_volatile_internals(&mut self) {
self.poison();
let (evt_tx, evt_rx) = flume::unbounded();
self.events = evt_tx;
spawn(async move {
trace!("Event processor restarted.");
super::events::runner(evt_rx).await;
trace!("Event processor finished.");
});
// Make mixer aware of new targets...
drop(
self.mixer
.send(MixerMessage::ReplaceInterconnect(self.clone())),
);
}
}

View File

@@ -0,0 +1,18 @@
#![allow(missing_docs)]
use super::Interconnect;
use crate::driver::DecodeConfig;
use dashmap::{DashMap, DashSet};
use serenity_voice_model::id::UserId;
pub enum UdpRxMessage {
SetConfig(DecodeConfig),
ReplaceInterconnect(Interconnect),
}
#[derive(Debug, Default)]
pub struct SsrcTracker {
pub disconnected_users: DashSet<UserId>,
pub user_ssrc_map: DashMap<UserId, u32>,
pub ssrc_user_map: DashMap<u32, UserId>,
}

View File

@@ -0,0 +1,12 @@
#![allow(missing_docs)]
use super::Interconnect;
use crate::{model::Event as GatewayEvent, ws::WsStream};
pub enum WsMessage {
Ws(Box<WsStream>),
ReplaceInterconnect(Interconnect),
SetKeepalive(f64),
Speaking(bool),
Deliver(GatewayEvent),
}

View File

@@ -0,0 +1,446 @@
use super::*;
/// Mix a track's audio stream into either the shared mixing buffer, or directly into the output
/// packet ("passthrough") when possible.
///
/// Passthrough is highest performance, but the source MUST be opus, have 20ms frames, and be the only
/// live track. In this case we copy the opus-encoded data with no changes. Otherwise, we fall back to
/// below.
///
/// There are a few functional requirements here for non-passthrough mixing that make it tricky:
/// * Input frame lengths are not congruent with what we need to send (i.e., 26.12ms in MP3 vs
/// needed 20ms).
/// * Input audio arrives at a different sample rate from required (i.e., 44.1 vs needed 48 kHz).
/// * Input data may not be `f32`s.
/// * Input data may not match stereo/mono of desired output.
///
/// All of the above challenges often happen at once. The rough pipeline in processing is:
///
/// until source end or 20 ms taken:
/// (use previous frame 'til empty / get new frame) -> [resample] -> [audio += vol * (sample as f32)]
///
/// Typically, we mix between a subset of the input packet and the output buf because the 20ms window
/// straddles packet boundaries. If there's enough space AND 48kHz AND receive f32s, then we use a fast
/// path.
///
/// In the mono -> stereo case, we duplicate across all target channels. In stereo -> mono, we average
/// the samples from each channel.
///
/// To avoid needing to hold onto resampled data longer than one mix cycle, we take enough input samples
/// to fill a chunk of the mixer (e.g., 10ms == 20ms / 2) so that they will all be used.
///
/// This is a fairly annoying piece of code to reason about, mainly because you need to hold so many
/// internal positions into: the mix buffer, resample buffers, and previous/current packets
/// for a stream.
#[inline]
pub fn mix_symph_indiv(
// shared buffer to mix into.
symph_mix: &mut AudioBuffer<f32>,
// buffer to hold built up packet
resample_scratch: &mut AudioBuffer<f32>,
// the input stream to use
input: &mut Parsed,
// resampler state and positions into partially read packets
local_state: &mut DecodeState,
// volume of this source
volume: f32,
// window into the output UDP buffer to copy opus frames into.
// This is set to `Some` IF passthrough is possible (i.e., one live source).
mut opus_slot: Option<&mut [u8]>,
) -> (MixType, MixStatus) {
let mut samples_written = 0;
let mut resample_in_progress = false;
let mut track_status = MixStatus::Live;
let codec_type = input.decoder.codec_params().codec;
resample_scratch.clear();
while samples_written != MONO_FRAME_SIZE {
// fetch a packet: either in progress, passthrough (early exit), or
let source_packet = if local_state.inner_pos != 0 {
Some(input.decoder.last_decoded())
} else if let Ok(pkt) = input.format.next_packet() {
if pkt.track_id() != input.track_id {
continue;
}
let buf = pkt.buf();
// Opus packet passthrough special case.
if codec_type == CODEC_TYPE_OPUS && local_state.passthrough != Passthrough::Block {
if let Some(slot) = opus_slot.as_mut() {
let sample_ct = if buf.is_empty() || buf.len() > i32::MAX as usize {
None
} else {
Some(opus2::packet::get_nb_samples(buf, SAMPLE_RATE))
};
// We don't actually block passthrough until a few violations are
// seen. The main one is that most Opus tracks end on a sub-20ms
// frame, particularly on Youtube.
// However, a frame that's bigger than the target buffer is an instant block.
let buf_size_fatal = buf.len() >= slot.len();
if match sample_ct {
Some(Ok(MONO_FRAME_SIZE)) => true,
_ => !local_state.record_and_check_passthrough_strike_final(buf_size_fatal),
} {
slot.write_all(buf)
.expect("Bounds check performed, and failure will block passthrough.");
return (MixType::Passthrough(buf.len()), MixStatus::Live);
}
}
}
input
.decoder
.decode(&pkt)
.map_err(|e| {
track_status = e.into();
})
.ok()
} else {
track_status = MixStatus::Ended;
None
};
// Cleanup: failed to get the next packet, but still have to convert and mix scratch.
if source_packet.is_none() {
if resample_in_progress {
// fill up remainder of buf with zeroes, resample, mix
let (chan_c, resampler, rs_out_buf) = local_state.resampler.as_mut().unwrap();
let in_len = resample_scratch.frames();
let to_render = resampler.input_frames_next().saturating_sub(in_len);
if to_render != 0 {
resample_scratch.render_reserved(Some(to_render));
for plane in resample_scratch.planes_mut().planes() {
for val in &mut plane[in_len..] {
*val = 0.0f32;
}
}
}
// Luckily, we make use of the WHOLE input buffer here.
resampler
.process_into_buffer(
&resample_scratch.planes().planes()[..*chan_c],
rs_out_buf,
None,
)
.unwrap();
// Calculate true end position using sample rate math
let ratio = (rs_out_buf[0].len() as f32) / (resample_scratch.frames() as f32);
let out_samples = (ratio * (in_len as f32)).round() as usize;
mix_resampled(rs_out_buf, symph_mix, samples_written, volume);
samples_written += out_samples;
}
break;
}
let source_packet = source_packet.unwrap();
let in_rate = source_packet.spec().rate;
let pkt_frames = source_packet.frames();
if pkt_frames == 0 {
continue;
}
if in_rate == SAMPLE_RATE_RAW as u32 {
// No need to resample: mix as standard.
let samples_marched = mix_over_ref(
&source_packet,
symph_mix,
local_state.inner_pos,
samples_written,
volume,
);
samples_written += samples_marched;
local_state.inner_pos += samples_marched;
local_state.inner_pos %= pkt_frames;
} else {
// NOTE: this should NEVER change in one stream.
let chan_c = source_packet.spec().channels.count();
let (_, resampler, rs_out_buf) = local_state.resampler.get_or_insert_with(|| {
// TODO: integ. error handling here.
let resampler = FftFixedOut::new(
in_rate as usize,
SAMPLE_RATE_RAW,
RESAMPLE_OUTPUT_FRAME_SIZE,
4,
chan_c,
)
.expect("Failed to create resampler.");
let out_buf = resampler.output_buffer_allocate(true);
(chan_c, resampler, out_buf)
});
let inner_pos = local_state.inner_pos;
let needed_in_frames = resampler.input_frames_next();
let available_frames = pkt_frames - inner_pos;
let force_copy = resample_in_progress || needed_in_frames > available_frames;
if (!force_copy) && matches!(source_packet, AudioBufferRef::F32(_)) {
// This is the only case where we can pull off a straight resample...
// I would really like if this could be a slice of slices,
// but the technology just isn't there yet. And I don't feel like
// writing unsafe transformations to do so.
// NOTE: if let needed as if-let && {bool} is nightly only.
if let AudioBufferRef::F32(s_pkt) = source_packet {
let refs: Vec<&[f32]> = s_pkt
.planes()
.planes()
.iter()
.map(|s| &s[inner_pos..][..needed_in_frames])
.collect();
local_state.inner_pos += needed_in_frames;
local_state.inner_pos %= pkt_frames;
resampler
.process_into_buffer(&refs, rs_out_buf, None)
.unwrap();
} else {
unreachable!()
}
} else {
// We either lack enough samples, or have the wrong data format, forcing
// a conversion/copy into the buffer.
let old_scratch_len = resample_scratch.frames();
let missing_frames = needed_in_frames - old_scratch_len;
let frames_to_take = available_frames.min(missing_frames);
resample_scratch.render_reserved(Some(frames_to_take));
copy_into_resampler(
&source_packet,
resample_scratch,
inner_pos,
old_scratch_len,
frames_to_take,
);
local_state.inner_pos += frames_to_take;
local_state.inner_pos %= pkt_frames;
if resample_scratch.frames() == needed_in_frames {
resampler
.process_into_buffer(
&resample_scratch.planes().planes()[..chan_c],
rs_out_buf,
None,
)
.unwrap();
resample_scratch.clear();
resample_in_progress = false;
} else {
// Not enough data to fill the resampler: fetch more.
resample_in_progress = true;
continue;
}
}
let samples_marched = mix_resampled(rs_out_buf, symph_mix, samples_written, volume);
samples_written += samples_marched;
}
}
(MixType::MixedPcm(samples_written), track_status)
}
#[inline]
fn mix_over_ref(
source: &AudioBufferRef<'_>,
target: &mut AudioBuffer<f32>,
source_pos: usize,
dest_pos: usize,
volume: f32,
) -> usize {
match source {
AudioBufferRef::U8(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::U16(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::U24(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::U32(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::S8(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::S16(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::S24(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::S32(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::F32(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
AudioBufferRef::F64(v) => mix_symph_buffer(v, target, source_pos, dest_pos, volume),
}
}
#[inline]
fn mix_symph_buffer<S>(
source: &AudioBuffer<S>,
target: &mut AudioBuffer<f32>,
source_pos: usize,
dest_pos: usize,
volume: f32,
) -> usize
where
S: Sample + IntoSample<f32>,
{
// mix in source_packet[inner_pos..] til end of EITHER buffer.
let src_usable = source.frames() - source_pos;
let tgt_usable = target.frames() - dest_pos;
let mix_ct = src_usable.min(tgt_usable);
let target_chans = target.spec().channels.count();
let target_mono = target_chans == 1;
let source_chans = source.spec().channels.count();
let source_mono = source_chans == 1;
let source_planes = source.planes();
let source_raw_planes = source_planes.planes();
if source_mono {
// mix this signal into *all* output channels at req'd volume.
let source_plane = source_raw_planes[0];
for d_plane in &mut (*target.planes_mut().planes()) {
for (d, s) in d_plane[dest_pos..dest_pos + mix_ct]
.iter_mut()
.zip(source_plane[source_pos..source_pos + mix_ct].iter())
{
*d += volume * (*s).into_sample();
}
}
} else if target_mono {
// mix all signals into the one target channel: reduce aggregate volume
// by n_channels.
let vol_adj = 1.0 / (source_chans as f32);
let mut t_planes = target.planes_mut();
let d_plane = &mut *t_planes.planes()[0];
for s_plane in source_raw_planes {
for (d, s) in d_plane[dest_pos..dest_pos + mix_ct]
.iter_mut()
.zip(s_plane[source_pos..source_pos + mix_ct].iter())
{
*d += volume * vol_adj * (*s).into_sample();
}
}
} else {
// stereo -> stereo: don't change volume, map input -> output channels w/ no duplication
for (d_plane, s_plane) in (*target.planes_mut().planes())
.iter_mut()
.zip(source_raw_planes[..].iter())
{
for (d, s) in d_plane[dest_pos..dest_pos + mix_ct]
.iter_mut()
.zip(s_plane[source_pos..source_pos + mix_ct].iter())
{
*d += volume * (*s).into_sample();
}
}
}
mix_ct
}
#[inline]
fn mix_resampled(
source: &[Vec<f32>],
target: &mut AudioBuffer<f32>,
dest_pos: usize,
volume: f32,
) -> usize {
let mix_ct = source[0].len();
let target_chans = target.spec().channels.count();
let target_mono = target_chans == 1;
let source_chans = source.len();
let source_mono = source_chans == 1;
// see `mix_symph_buffer` for explanations of stereo<->mono logic.
if source_mono {
let source_plane = &source[0];
for d_plane in &mut (*target.planes_mut().planes()) {
for (d, s) in d_plane[dest_pos..dest_pos + mix_ct]
.iter_mut()
.zip(source_plane)
{
*d += volume * s;
}
}
} else if target_mono {
let vol_adj = 1.0 / (source_chans as f32);
let mut t_planes = target.planes_mut();
let d_plane = &mut *t_planes.planes()[0];
for s_plane in source {
for (d, s) in d_plane[dest_pos..dest_pos + mix_ct].iter_mut().zip(s_plane) {
*d += volume * vol_adj * s;
}
}
} else {
for (d_plane, s_plane) in (*target.planes_mut().planes())
.iter_mut()
.zip(source[..].iter())
{
for (d, s) in d_plane[dest_pos..dest_pos + mix_ct].iter_mut().zip(s_plane) {
*d += volume * (*s);
}
}
}
mix_ct
}
#[inline]
pub(crate) fn copy_into_resampler(
source: &AudioBufferRef<'_>,
target: &mut AudioBuffer<f32>,
source_pos: usize,
dest_pos: usize,
len: usize,
) -> usize {
match source {
AudioBufferRef::U8(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::U16(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::U24(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::U32(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::S8(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::S16(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::S24(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::S32(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::F32(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
AudioBufferRef::F64(v) => copy_symph_buffer(v, target, source_pos, dest_pos, len),
}
}
#[inline]
fn copy_symph_buffer<S>(
source: &AudioBuffer<S>,
target: &mut AudioBuffer<f32>,
source_pos: usize,
dest_pos: usize,
len: usize,
) -> usize
where
S: Sample + IntoSample<f32>,
{
for (d_plane, s_plane) in (*target.planes_mut().planes())
.iter_mut()
.zip(source.planes().planes()[..].iter())
{
for (d, s) in d_plane[dest_pos..dest_pos + len]
.iter_mut()
.zip(s_plane[source_pos..source_pos + len].iter())
{
*d = (*s).into_sample();
}
}
len
}

View File

@@ -0,0 +1,884 @@
pub mod mix_logic;
mod pool;
mod result;
pub mod state;
pub mod track;
mod util;
use pool::*;
use result::*;
use state::*;
pub use track::*;
use super::{
disposal::DisposalThread,
error::{Error, Result},
message::*,
};
use crate::{
constants::*,
driver::{CryptoMode, MixMode},
events::EventStore,
input::{Input, Parsed},
tracks::{Action, LoopState, PlayError, PlayMode, TrackCommand, TrackHandle, TrackState, View},
Config,
};
use discortp::{
discord::MutableKeepalivePacket,
rtp::{MutableRtpPacket, RtpPacket},
MutablePacket,
};
use flume::{Receiver, SendError, Sender, TryRecvError};
use opus2::{Application as CodingMode, Bitrate, Encoder as OpusEncoder, SoftClip};
use rand::random;
use rubato::{FftFixedOut, Resampler};
use std::{
io::Write,
result::Result as StdResult,
sync::{atomic::Ordering, Arc},
time::{Duration, Instant},
};
use symphonia_core::{
audio::{AudioBuffer, AudioBufferRef, Layout, SampleBuffer, Signal, SignalSpec},
codecs::CODEC_TYPE_OPUS,
conv::IntoSample,
formats::SeekTo,
sample::Sample,
units::Time,
};
use tokio::runtime::Handle;
use tracing::error;
#[cfg(test)]
use crate::driver::test_config::{OutputMessage, OutputMode};
#[cfg(test)]
use discortp::Packet as _;
pub struct Mixer {
pub bitrate: Bitrate,
pub config: Arc<Config>,
pub conn_active: Option<MixerConnection>,
pub deadline: Instant,
pub disposer: DisposalThread,
pub encoder: OpusEncoder,
pub interconnect: Interconnect,
pub mix_rx: Receiver<MixerMessage>,
pub muted: bool,
// pub packet: [u8; VOICE_PACKET_MAX],
pub prevent_events: bool,
pub silence_frames: u8,
pub soft_clip: SoftClip,
thread_pool: BlockyTaskPool,
pub ws: Option<Sender<WsMessage>>,
pub keepalive_deadline: Instant,
pub keepalive_packet: [u8; MutableKeepalivePacket::minimum_packet_size()],
pub tracks: Vec<InternalTrack>,
track_handles: Vec<TrackHandle>,
sample_buffer: SampleBuffer<f32>,
symph_mix: AudioBuffer<f32>,
resample_scratch: AudioBuffer<f32>,
#[cfg(test)]
pub remaining_loops: Option<u64>,
#[cfg(test)]
raw_msg: Option<OutputMessage>,
}
fn new_encoder(bitrate: Bitrate, mix_mode: MixMode) -> Result<OpusEncoder> {
let mut encoder = OpusEncoder::new(SAMPLE_RATE, mix_mode.to_opus(), CodingMode::Audio)?;
encoder.set_bitrate(bitrate)?;
Ok(encoder)
}
impl Mixer {
#[must_use]
pub fn new(
mix_rx: Receiver<MixerMessage>,
async_handle: Handle,
interconnect: Interconnect,
config: Config,
) -> Self {
let bitrate = DEFAULT_BITRATE;
let encoder = new_encoder(bitrate, config.mix_mode)
.expect("Failed to create encoder in mixing thread with known-good values.");
let soft_clip = SoftClip::new(config.mix_mode.to_opus());
let keepalive_packet = [0u8; MutableKeepalivePacket::minimum_packet_size()];
let tracks = Vec::with_capacity(usize::from(1.max(config.preallocated_tracks)));
let track_handles = Vec::with_capacity(usize::from(1.max(config.preallocated_tracks)));
let thread_pool = BlockyTaskPool::new(async_handle);
let symph_layout = config.mix_mode.symph_layout();
let disposer = config.disposer.clone().unwrap_or_default();
let config = config.into();
let sample_buffer = SampleBuffer::<f32>::new(
MONO_FRAME_SIZE as u64,
symphonia_core::audio::SignalSpec::new_with_layout(
SAMPLE_RATE_RAW as u32,
symph_layout,
),
);
let symph_mix = AudioBuffer::<f32>::new(
MONO_FRAME_SIZE as u64,
symphonia_core::audio::SignalSpec::new_with_layout(
SAMPLE_RATE_RAW as u32,
symph_layout,
),
);
let resample_scratch = AudioBuffer::<f32>::new(
MONO_FRAME_SIZE as u64,
SignalSpec::new_with_layout(SAMPLE_RATE_RAW as u32, Layout::Stereo),
);
let deadline = Instant::now();
Self {
bitrate,
config,
conn_active: None,
deadline,
disposer,
encoder,
interconnect,
mix_rx,
muted: false,
prevent_events: false,
silence_frames: 0,
soft_clip,
thread_pool,
ws: None,
keepalive_deadline: deadline,
keepalive_packet,
tracks,
track_handles,
sample_buffer,
symph_mix,
resample_scratch,
#[cfg(test)]
remaining_loops: None,
#[cfg(test)]
raw_msg: None,
}
}
fn set_bitrate(&mut self, bitrate: Bitrate) -> Result<()> {
self.encoder.set_bitrate(bitrate).map_err(Into::into)
}
pub(crate) fn do_rebuilds(
&mut self,
event_failure: bool,
conn_failure: bool,
) -> StdResult<(), SendError<CoreMessage>> {
// event failure? rebuild interconnect.
// ws or udp failure? full connect
// (soft reconnect is covered by the ws task.)
//
// in both cases, send failure is fatal,
// but will only occur on disconnect.
if event_failure {
self.rebuild_interconnect()?;
}
if conn_failure {
self.full_reconnect_gateway()?;
}
Ok(())
}
pub(crate) fn rebuild_interconnect(&mut self) -> StdResult<(), SendError<CoreMessage>> {
self.prevent_events = true;
self.interconnect
.core
.send(CoreMessage::RebuildInterconnect)
}
pub(crate) fn full_reconnect_gateway(&mut self) -> StdResult<(), SendError<CoreMessage>> {
self.conn_active = None;
self.interconnect.core.send(CoreMessage::FullReconnect)
}
#[inline]
pub(crate) fn handle_message(
&mut self,
msg: MixerMessage,
packet: &mut [u8],
) -> (bool, bool, bool) {
let mut events_failure = false;
let mut conn_failure = false;
let mut should_exit = false;
let error = match msg {
MixerMessage::AddTrack(t) => self.add_track(*t),
MixerMessage::SetTrack(t) => {
self.tracks.clear();
self.track_handles.clear();
let mut out = self.fire_event(EventMessage::RemoveAllTracks);
if let Some(t) = t {
// Do this unconditionally: this affects local state infallibly,
// with the event installation being the remote part.
if let Err(e) = self.add_track(*t) {
out = Err(e);
}
}
out
},
MixerMessage::SetBitrate(b) => {
self.bitrate = b;
if let Err(e) = self.set_bitrate(b) {
error!("Failed to update bitrate {:?}", e);
}
Ok(())
},
MixerMessage::SetMute(m) => {
self.muted = m;
Ok(())
},
MixerMessage::SetConn(conn, ssrc) => {
self.conn_active = Some(conn);
let mut rtp = MutableRtpPacket::new(packet).expect(
"Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
rtp.set_ssrc(ssrc);
rtp.set_sequence(random::<u16>().into());
rtp.set_timestamp(random::<u32>().into());
self.deadline = Instant::now();
self.update_keepalive(ssrc);
Ok(())
},
MixerMessage::DropConn => {
self.conn_active = None;
Ok(())
},
MixerMessage::ReplaceInterconnect(i) => {
self.prevent_events = false;
if let Some(ws) = &self.ws {
conn_failure |= ws.send(WsMessage::ReplaceInterconnect(i.clone())).is_err();
}
#[cfg(feature = "receive")]
if let Some(conn) = &self.conn_active {
conn_failure |= conn
.udp_rx
.send(UdpRxMessage::ReplaceInterconnect(i.clone()))
.is_err();
}
self.interconnect = i;
self.rebuild_tracks()
},
MixerMessage::SetConfig(new_config) => {
if new_config.mix_mode != self.config.mix_mode {
self.soft_clip = SoftClip::new(new_config.mix_mode.to_opus());
if let Ok(enc) = new_encoder(self.bitrate, new_config.mix_mode) {
self.encoder = enc;
} else {
self.bitrate = DEFAULT_BITRATE;
self.encoder = new_encoder(self.bitrate, new_config.mix_mode)
.expect("Failed fallback rebuild of OpusEncoder with safe inputs.");
}
let sl = new_config.mix_mode.symph_layout();
self.sample_buffer = SampleBuffer::<f32>::new(
MONO_FRAME_SIZE as u64,
SignalSpec::new_with_layout(SAMPLE_RATE_RAW as u32, sl),
);
self.symph_mix = AudioBuffer::<f32>::new(
MONO_FRAME_SIZE as u64,
SignalSpec::new_with_layout(SAMPLE_RATE_RAW as u32, sl),
);
}
self.config = Arc::new(
#[cfg(feature = "receive")]
new_config.clone(),
#[cfg(not(feature = "receive"))]
new_config,
);
let preallocated_tracks = usize::from(self.config.preallocated_tracks);
if self.tracks.capacity() < preallocated_tracks {
self.tracks.reserve(preallocated_tracks - self.tracks.len());
}
#[cfg(feature = "receive")]
if let Some(conn) = &self.conn_active {
if let crate::driver::DecodeMode::Decode(decode_config) = new_config.decode_mode
{
let msg = UdpRxMessage::SetConfig(decode_config);
conn_failure |= conn.udp_rx.send(msg).is_err();
}
}
Ok(())
},
MixerMessage::RebuildEncoder => match new_encoder(self.bitrate, self.config.mix_mode) {
Ok(encoder) => {
self.encoder = encoder;
Ok(())
},
Err(e) => {
error!("Failed to rebuild encoder. Resetting bitrate. {:?}", e);
self.bitrate = DEFAULT_BITRATE;
self.encoder = new_encoder(self.bitrate, self.config.mix_mode)
.expect("Failed fallback rebuild of OpusEncoder with safe inputs.");
Ok(())
},
},
MixerMessage::Ws(new_ws_handle) => {
self.ws = new_ws_handle;
if let Err(e) = self.send_gateway_speaking() {
conn_failure |= e.should_trigger_connect();
}
Ok(())
},
MixerMessage::Poison => {
should_exit = true;
Ok(())
},
};
if let Err(e) = error {
events_failure |= e.should_trigger_interconnect_rebuild();
conn_failure |= e.should_trigger_connect();
}
(events_failure, conn_failure, should_exit)
}
pub(crate) fn update_keepalive(&mut self, ssrc: u32) {
let mut ka = MutableKeepalivePacket::new(&mut self.keepalive_packet[..])
.expect("FATAL: Insufficient bytes given to keepalive packet.");
ka.set_ssrc(ssrc);
self.keepalive_deadline = self.deadline + UDP_KEEPALIVE_GAP;
}
#[inline]
pub(crate) fn fire_event(&self, event: EventMessage) -> Result<()> {
// As this task is responsible for noticing the potential death of an event context,
// it's responsible for not forcibly recreating said context repeatedly.
if !self.prevent_events {
self.interconnect.events.send(event)?;
}
Ok(())
}
#[inline]
pub fn add_track(&mut self, track: TrackContext) -> Result<()> {
let (track, evts, state, handle) = InternalTrack::decompose_track(track);
self.tracks.push(track);
self.track_handles.push(handle.clone());
self.interconnect
.events
.send(EventMessage::AddTrack(evts, state, handle))?;
Ok(())
}
// rebuilds the event thread's view of each track, in event of a full rebuild.
#[inline]
fn rebuild_tracks(&mut self) -> Result<()> {
for (track, handle) in self.tracks.iter().zip(self.track_handles.iter()) {
let evts = EventStore::default();
let state = track.state();
let handle = handle.clone();
self.interconnect
.events
.send(EventMessage::AddTrack(evts, state, handle))?;
}
Ok(())
}
#[inline]
pub(crate) fn audio_commands_events(&mut self) -> Result<()> {
// Apply user commands.
for (i, track) in self.tracks.iter_mut().enumerate() {
// This causes fallible event system changes,
// but if the event thread has died then we'll certainly
// detect that on the tick later.
// Changes to play state etc. MUST all be handled.
let action = track.process_commands(i, &self.interconnect);
if let Some(req) = action.seek_point {
track.seek(
i,
req,
&self.interconnect,
&self.thread_pool,
&self.config,
self.prevent_events,
);
}
if let Some(callback) = action.make_playable {
if let Err(e) = track.get_or_ready_input(
i,
&self.interconnect,
&self.thread_pool,
&self.config,
self.prevent_events,
) {
track.callbacks.make_playable = Some(callback);
if let Some(fail) = e.as_user() {
track.playing = PlayMode::Errored(fail);
}
if let Some(req) = e.into_seek_request() {
track.seek(
i,
req,
&self.interconnect,
&self.thread_pool,
&self.config,
self.prevent_events,
);
}
} else {
// Track is already ready: don't register callback and just act.
drop(callback.send(Ok(())));
}
}
}
let mut i = 0;
while i < self.tracks.len() {
let track = self
.tracks
.get_mut(i)
.expect("Tried to remove an illegal track index.");
if track.playing.is_done() {
let p_state = track.playing.clone();
let to_drop = self.tracks.swap_remove(i);
self.disposer
.dispose(DisposalMessage::Track(Box::new(to_drop)));
let to_drop = self.track_handles.swap_remove(i);
self.disposer.dispose(DisposalMessage::Handle(to_drop));
self.fire_event(EventMessage::ChangeState(
i,
TrackStateChange::Mode(p_state),
))?;
} else {
i += 1;
}
}
// Tick -- receive side also handles removals in same manner after it increments
// times etc.
self.fire_event(EventMessage::Tick)?;
Ok(())
}
#[cfg(test)]
#[inline]
pub(crate) fn test_signal_empty_tick(&self) {
match &self.config.override_connection {
Some(OutputMode::Raw(tx)) =>
drop(tx.send(crate::driver::test_config::TickMessage::NoEl)),
Some(OutputMode::Rtp(tx)) =>
drop(tx.send(crate::driver::test_config::TickMessage::NoEl)),
None => {},
}
}
pub fn crypto_mode(&self) -> CryptoMode {
let mode = self.conn_active.as_ref().map(|v| v.crypto_state.kind());
if cfg!(not(test)) {
mode.expect("Shouldn't be mixing packets without access to a cipher + UDP dest.")
} else {
mode.unwrap_or_else(|| self.config.crypto_mode)
}
}
#[inline]
pub fn mix_and_build_packet(&mut self, packet: &mut [u8]) -> Result<usize> {
// symph_mix is an `AudioBuffer` (planar format), we need to convert this
// later into an interleaved `SampleBuffer` for libopus.
self.symph_mix.clear();
self.symph_mix.render_reserved(Some(MONO_FRAME_SIZE));
self.resample_scratch.clear();
// Walk over all the audio files, combining into one audio frame according
// to volume, play state, etc.
let mut mix_len = {
let out = self.mix_tracks(packet);
self.sample_buffer.copy_interleaved_typed(&self.symph_mix);
out
};
if self.muted {
mix_len = MixType::MixedPcm(0);
}
// Explicit "Silence" frame handling: if there is no mixed data, we must send
// ~5 frames of silence (unless another good audio frame appears) before we
// stop sending RTP frames.
if mix_len == MixType::MixedPcm(0) {
if self.silence_frames > 0 {
self.silence_frames -= 1;
let mut rtp = MutableRtpPacket::new(packet).expect(
"FATAL: Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
let payload = rtp.payload_mut();
let pre_len = self.crypto_mode().payload_prefix_len();
payload[pre_len..pre_len + SILENT_FRAME.len()].copy_from_slice(&SILENT_FRAME[..]);
mix_len = MixType::Passthrough(SILENT_FRAME.len());
} else {
// Per official guidelines, send 5x silence BEFORE we stop speaking.
return Ok(0);
}
} else {
self.silence_frames = 5;
if let MixType::MixedPcm(n) = mix_len {
if self.config.use_softclip {
self.soft_clip.apply(
&mut self.sample_buffer.samples_mut()
[..n * self.config.mix_mode.channels()],
);
}
}
}
// For the benefit of test cases, send the raw un-RTP'd data.
#[cfg(test)]
let out = if let Some(OutputMode::Raw(_)) = &self.config.override_connection {
// This case has been handled before buffer clearing above.
let msg = match mix_len {
MixType::Passthrough(len) if len == SILENT_FRAME.len() => OutputMessage::Silent,
MixType::Passthrough(len) => {
let rtp = RtpPacket::new(packet).expect(
"FATAL: Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
let payload = rtp.payload();
let opus_frame =
(payload[self.crypto_mode().payload_prefix_len()..][..len]).to_vec();
OutputMessage::Passthrough(opus_frame)
},
MixType::MixedPcm(_) => OutputMessage::Mixed(
self.sample_buffer.samples()[..self.config.mix_mode.sample_count_in_frame()]
.to_vec(),
),
};
self.raw_msg = Some(msg);
Ok(1)
} else {
self.prep_packet(mix_len, packet)
};
#[cfg(not(test))]
let out = self.prep_packet(mix_len, packet);
// Zero out all planes of the mix buffer if any audio was written.
if matches!(mix_len, MixType::MixedPcm(a) if a > 0) {
for plane in self.symph_mix.planes_mut().planes() {
plane.fill(0.0);
}
}
out
}
#[inline]
fn prep_packet(&mut self, mix_len: MixType, packet: &mut [u8]) -> Result<usize> {
let send_buffer = self.sample_buffer.samples();
let conn = self
.conn_active
.as_mut()
.expect("Shouldn't be mixing packets without access to a cipher + UDP dest.");
let mut rtp = MutableRtpPacket::new(packet).expect(
"FATAL: Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
let payload = rtp.payload_mut();
let crypto_mode = conn.crypto_state.kind();
let first_payload_byte = crypto_mode.payload_prefix_len();
let total_payload_space = payload.len() - crypto_mode.payload_suffix_len();
// If passthrough, Opus payload in place already.
// Else encode into buffer with space for AEAD encryption headers.
let mut payload_len = match mix_len {
MixType::Passthrough(opus_len) => opus_len,
MixType::MixedPcm(_samples) => self.encoder.encode_float(
&send_buffer[..self.config.mix_mode.sample_count_in_frame()],
&mut payload[first_payload_byte..total_payload_space],
)?,
};
if conn.dave_protocol_version.load(Ordering::Relaxed) != 0 {
if let Some(ref mut dave_session) = *conn.dave_session.write().unwrap() {
if dave_session.is_ready() {
let encrypted = dave_session
.encrypt_opus(
&payload[first_payload_byte..first_payload_byte + payload_len],
)?
.into_owned();
payload_len = encrypted.len();
payload[first_payload_byte..first_payload_byte + payload_len]
.copy_from_slice(&encrypted);
}
}
}
let final_payload_size = conn
.crypto_state
.write_packet_nonce(&mut rtp, first_payload_byte + payload_len);
// Packet encryption ignored in test modes.
#[cfg(not(test))]
let encrypt = true;
#[cfg(test)]
let encrypt = self.config.override_connection.is_none();
if encrypt {
conn.cipher
.encrypt_pkt_in_place(&mut rtp, final_payload_size)?;
}
Ok(RtpPacket::minimum_packet_size() + final_payload_size)
}
#[inline]
pub(crate) fn send_packet(&self, packet: &[u8]) -> Result<()> {
#[cfg(test)]
let send_status = if let Some(OutputMode::Raw(tx)) = &self.config.override_connection {
// This case has been handled before buffer clearing in `mix_and_build_packet`.
drop(tx.send(self.raw_msg.clone().unwrap().into()));
Ok(())
} else {
self.send_packet_(packet)
};
#[cfg(not(test))]
let send_status = self.send_packet_(packet);
send_status.or_else(Error::disarm_would_block)?;
Ok(())
}
#[inline]
fn send_packet_(&self, packet: &[u8]) -> Result<()> {
let conn = self
.conn_active
.as_ref()
.expect("Shouldn't be mixing packets without access to a cipher + UDP dest.");
#[cfg(test)]
if let Some(OutputMode::Rtp(tx)) = &self.config.override_connection {
// Test mode: send unencrypted (compressed) packets to local receiver.
drop(tx.send(packet.to_vec().into()));
} else {
conn.udp_tx.send(packet)?;
}
#[cfg(not(test))]
{
// Normal operation: send encrypted payload to UDP Tx task.
conn.udp_tx.send(packet)?;
}
Ok(())
}
#[inline]
pub(crate) fn check_and_send_keepalive(&mut self, now: Option<Instant>) -> Result<()> {
if let Some(conn) = self.conn_active.as_mut() {
let now = now.unwrap_or_else(Instant::now);
if now >= self.keepalive_deadline {
conn.udp_tx.send(&self.keepalive_packet)?;
self.keepalive_deadline += UDP_KEEPALIVE_GAP;
}
}
Ok(())
}
#[inline]
pub(crate) fn send_gateway_speaking(&self) -> Result<()> {
if let Some(ws) = &self.ws {
ws.send(WsMessage::Speaking(true))?;
}
Ok(())
}
#[inline]
pub(crate) fn send_gateway_not_speaking(&self) {
if let Some(ws) = &self.ws {
// NOTE: this explicit `drop` should prevent a catastrophic thread pileup.
// A full reconnect might cause an inner closed connection.
// It's safer to leave the central task to clean this up and
// pass the mixer a new channel.
drop(ws.send(WsMessage::Speaking(false)));
}
}
#[inline]
fn mix_tracks(&mut self, packet: &mut [u8]) -> MixType {
// Get a slice of bytes to write in data for Opus packet passthrough.
let mut rtp = MutableRtpPacket::new(packet).expect(
"FATAL: Too few bytes in self.packet for RTP header.\
(Blame: VOICE_PACKET_MAX?)",
);
let payload = rtp.payload_mut();
let opus_frame = &mut payload[self.crypto_mode().payload_prefix_len()..];
// Opus frame passthrough.
// This requires that we have only one PLAYING track, who has volume 1.0, and an
// Opus codec type (verified later in mix_symph_indiv).
//
// We *could* cache the number of live tracks separately, but that makes this
// quite fragile given all the ways a user can alter the PlayMode.
let mut num_live = 0;
let mut last_live_vol = 1.0;
for track in &self.tracks {
if track.playing.is_playing() {
num_live += 1;
last_live_vol = track.volume;
}
}
let do_passthrough = num_live == 1 && (last_live_vol - 1.0).abs() < f32::EPSILON;
let mut len = 0;
for (i, track) in self.tracks.iter_mut().enumerate() {
let vol = track.volume;
// This specifically tries to get tracks who are "preparing",
// so that event handlers and the like can all be fired without
// the track being in a `Play` state.
if !track.should_check_input() {
continue;
}
let should_play = track.playing.is_playing();
let input = track.get_or_ready_input(
i,
&self.interconnect,
&self.thread_pool,
&self.config,
self.prevent_events,
);
let (input, mix_state) = match input {
Ok(i) => i,
Err(InputReadyingError::Waiting) => continue,
Err(InputReadyingError::NeedsSeek(req)) => {
track.seek(
i,
req,
&self.interconnect,
&self.thread_pool,
&self.config,
self.prevent_events,
);
continue;
},
// TODO: allow for retry in given time.
Err(e) => {
if let Some(fail) = e.as_user() {
track.playing = PlayMode::Errored(fail);
}
continue;
},
};
// Now that we have dealt with potential errors in preparing tracks,
// only do any mixing if the track is to be played!
if !should_play {
continue;
}
let (mix_type, status) = mix_logic::mix_symph_indiv(
&mut self.symph_mix,
&mut self.resample_scratch,
input,
mix_state,
vol,
do_passthrough.then_some(&mut *opus_frame),
);
let return_here = if let MixType::MixedPcm(pcm_len) = mix_type {
len = len.max(pcm_len);
false
} else {
if mix_state.passthrough == Passthrough::Inactive {
input.decoder.reset();
}
mix_state.passthrough = Passthrough::Active;
true
};
// FIXME: allow Ended to trigger a seek/loop/revisit in the same mix cycle?
// Would this be possible with special-casing to mark some inputs as fast
// to recreate? Probably not doable in the general case.
match status {
MixStatus::Live => track.step_frame(),
MixStatus::Errored(e) =>
track.playing = PlayMode::Errored(PlayError::Decode(e.into())),
MixStatus::Ended if track.do_loop() => {
drop(self.track_handles[i].seek(Duration::default()));
if !self.prevent_events {
// position update is sent out later, when the seek concludes.
drop(self.interconnect.events.send(EventMessage::ChangeState(
i,
TrackStateChange::Loops(track.loops, false),
)));
}
},
MixStatus::Ended => {
track.end();
},
}
// This needs to happen here due to borrow checker shenanigans.
if return_here {
return mix_type;
}
}
MixType::MixedPcm(len)
}
}

View File

@@ -0,0 +1,141 @@
use super::util::copy_seek_to;
use crate::{
driver::tasks::message::MixerInputResultMessage,
input::{AudioStream, AudioStreamError, Compose, Input, LiveInput, Parsed},
Config,
};
use flume::Sender;
use rusty_pool::ThreadPool;
use std::{result::Result as StdResult, sync::Arc, time::Duration};
use symphonia_core::{
formats::{SeekMode, SeekTo},
io::MediaSource,
};
use tokio::runtime::Handle;
#[derive(Clone)]
pub struct BlockyTaskPool {
pool: ThreadPool,
handle: Handle,
}
impl BlockyTaskPool {
pub fn new(handle: Handle) -> Self {
Self {
pool: ThreadPool::new(0, 64, Duration::from_secs(5)),
handle,
}
}
pub fn create(
&self,
callback: Sender<MixerInputResultMessage>,
input: Input,
seek_time: Option<SeekTo>,
config: Arc<Config>,
) {
// Moves an Input from Lazy -> Live.
// We either do this on this pool, or move it to the tokio executor as the source requires.
// This takes a seek_time to pass on and execute *after* parsing (i.e., back-seek on
// read-only stream).
match input {
Input::Lazy(mut lazy) => {
let far_pool = self.clone();
if lazy.should_create_async() {
self.handle.spawn(async move {
let out = lazy.create_async().await;
far_pool.send_to_parse(out, lazy, callback, seek_time, config);
});
} else {
self.pool.execute(move || {
let out = lazy.create();
far_pool.send_to_parse(out, lazy, callback, seek_time, config);
});
}
},
Input::Live(live, maybe_create) =>
self.parse(config, callback, live, maybe_create, seek_time),
}
}
pub fn send_to_parse(
&self,
create_res: StdResult<AudioStream<Box<dyn MediaSource>>, AudioStreamError>,
rec: Box<dyn Compose>,
callback: Sender<MixerInputResultMessage>,
seek_time: Option<SeekTo>,
config: Arc<Config>,
) {
match create_res {
Ok(o) => {
self.parse(config, callback, LiveInput::Raw(o), Some(rec), seek_time);
},
Err(e) => {
drop(callback.send(MixerInputResultMessage::CreateErr(e.into())));
},
}
}
pub fn parse(
&self,
config: Arc<Config>,
callback: Sender<MixerInputResultMessage>,
input: LiveInput,
rec: Option<Box<dyn Compose>>,
seek_time: Option<SeekTo>,
) {
let pool_clone = self.clone();
self.pool.execute(move || {
match input.promote(config.codec_registry, config.format_registry) {
Ok(LiveInput::Parsed(parsed)) => match seek_time {
// If seek time is zero, then wipe it out.
// Some formats (MKV) make SeekTo(0) require a backseek to realign with the
// current page.
Some(seek_time) if !super::util::seek_to_is_zero(&seek_time) => {
pool_clone.seek(callback, parsed, rec, seek_time, false, config);
},
_ => {
drop(callback.send(MixerInputResultMessage::Built(parsed, rec)));
},
},
Ok(_) => unreachable!(),
Err(e) => {
drop(callback.send(MixerInputResultMessage::ParseErr(e.into())));
},
}
});
}
pub fn seek(
&self,
callback: Sender<MixerInputResultMessage>,
mut input: Parsed,
rec: Option<Box<dyn Compose>>,
seek_time: SeekTo,
// Not all of symphonia's formats bother to return SeekErrorKind::ForwardOnly.
// So, we need *this* flag.
backseek_needed: bool,
config: Arc<Config>,
) {
let pool_clone = self.clone();
self.pool.execute(move || match rec {
Some(rec) if (!input.supports_backseek) && backseek_needed => {
pool_clone.create(callback, Input::Lazy(rec), Some(seek_time), config);
},
_ => {
let seek_result = input
.format
.seek(SeekMode::Accurate, copy_seek_to(&seek_time));
input.decoder.reset();
drop(callback.send(MixerInputResultMessage::Seek(
input,
rec,
seek_result.map_err(Arc::new),
)));
},
});
}
}

View File

@@ -0,0 +1,55 @@
use crate::{
input::AudioStreamError,
tracks::{PlayError, SeekRequest},
};
use std::sync::Arc;
use symphonia_core::errors::Error as SymphoniaError;
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
pub enum MixType {
Passthrough(usize),
MixedPcm(usize),
}
pub enum MixStatus {
Live,
Ended,
Errored(SymphoniaError),
}
impl From<SymphoniaError> for MixStatus {
fn from(e: SymphoniaError) -> Self {
Self::Errored(e)
}
}
// The Symph errors are Arc'd here since if they come up, they will always
// be Arc'd anyway via into_user.
#[derive(Clone, Debug)]
pub enum InputReadyingError {
Parsing(Arc<SymphoniaError>),
Creation(Arc<AudioStreamError>),
Seeking(Arc<SymphoniaError>),
Dropped,
Waiting,
NeedsSeek(SeekRequest),
}
impl InputReadyingError {
pub fn as_user(&self) -> Option<PlayError> {
match self {
Self::Parsing(e) => Some(PlayError::Parse(e.clone())),
Self::Creation(e) => Some(PlayError::Create(e.clone())),
Self::Seeking(e) => Some(PlayError::Seek(e.clone())),
_ => None,
}
}
pub fn into_seek_request(self) -> Option<SeekRequest> {
if let Self::NeedsSeek(a) = self {
Some(a)
} else {
None
}
}
}

View File

@@ -0,0 +1,105 @@
use crate::{
constants::OPUS_PASSTHROUGH_STRIKE_LIMIT,
driver::tasks::message::*,
input::{Compose, Input, LiveInput, Metadata, Parsed},
tracks::{ReadyState, SeekRequest},
};
use flume::Receiver;
use rubato::FftFixedOut;
use std::time::Instant;
pub enum InputState {
NotReady(Input),
Preparing(PreparingInfo),
Ready(Parsed, Option<Box<dyn Compose>>),
}
impl InputState {
pub fn metadata(&mut self) -> Option<Metadata<'_>> {
if let Self::Ready(parsed, _) = self {
Some(parsed.into())
} else {
None
}
}
#[must_use]
pub fn ready_state(&self) -> ReadyState {
match self {
Self::NotReady(_) => ReadyState::Uninitialised,
Self::Preparing(_) => ReadyState::Preparing,
Self::Ready(_, _) => ReadyState::Playable,
}
}
}
impl From<Input> for InputState {
fn from(val: Input) -> Self {
match val {
a @ Input::Lazy(_) => Self::NotReady(a),
Input::Live(live, rec) => match live {
LiveInput::Parsed(p) => Self::Ready(p, rec),
other => Self::NotReady(Input::Live(other, rec)),
},
}
}
}
pub struct PreparingInfo {
#[allow(dead_code)]
/// Time this request was fired.
pub time: Instant,
/// Used to handle seek requests fired while a track was being created (or a seek was in progress).
pub queued_seek: Option<SeekRequest>,
/// Callback from the thread pool to indicate the result of creating/parsing this track.
pub callback: Receiver<MixerInputResultMessage>,
}
pub struct DecodeState {
pub inner_pos: usize,
pub resampler: Option<(usize, FftFixedOut<f32>, Vec<Vec<f32>>)>,
pub passthrough: Passthrough,
pub passthrough_violations: u8,
}
impl DecodeState {
pub fn reset(&mut self) {
self.inner_pos = 0;
self.resampler = None;
}
pub fn record_and_check_passthrough_strike_final(&mut self, fatal: bool) -> bool {
self.passthrough_violations = self.passthrough_violations.saturating_add(1);
let blocked = fatal || self.passthrough_violations > OPUS_PASSTHROUGH_STRIKE_LIMIT;
if blocked {
self.passthrough = Passthrough::Block;
}
blocked
}
}
impl Default for DecodeState {
fn default() -> Self {
Self {
inner_pos: 0,
resampler: None,
passthrough: Passthrough::Inactive,
passthrough_violations: 0,
}
}
}
/// Simple state to manage decoder resets etc.
///
/// Inactive->Active transitions should trigger a reset.
///
/// Block should be used if a source contains known-bad packets:
/// it's unlikely that packet sizes will vary, but if they do then
/// we can't passthrough (and every attempt will trigger a codec reset,
/// which probably won't sound too smooth).
#[derive(Clone, Copy, Eq, PartialEq)]
pub enum Passthrough {
Active,
Inactive,
Block,
}

View File

@@ -0,0 +1,403 @@
use crate::tracks::{ReadyState, SeekRequest};
use std::result::Result as StdResult;
use symphonia_core::errors::Error as SymphError;
use super::*;
pub struct InternalTrack {
pub(crate) playing: PlayMode,
pub(crate) volume: f32,
pub(crate) input: InputState,
pub(crate) mix_state: DecodeState,
pub(crate) position: Duration,
pub(crate) play_time: Duration,
pub(crate) commands: Receiver<TrackCommand>,
pub(crate) loops: LoopState,
pub(crate) callbacks: Callbacks,
}
impl<'a> InternalTrack {
pub(crate) fn decompose_track(
val: TrackContext,
) -> (Self, EventStore, TrackState, TrackHandle) {
let TrackContext {
handle,
track,
receiver,
} = val;
let out = InternalTrack {
playing: track.playing,
volume: track.volume,
input: InputState::from(track.input),
mix_state: DecodeState::default(),
position: Duration::default(),
play_time: Duration::default(),
commands: receiver,
loops: track.loops,
callbacks: Callbacks::default(),
};
let state = out.state();
(out, track.events, state, handle)
}
pub(crate) fn state(&self) -> TrackState {
let ready = self.input.ready_state();
TrackState {
playing: self.playing.clone(),
volume: self.volume,
position: self.position,
play_time: self.play_time,
loops: self.loops,
ready,
}
}
pub(crate) fn view(&'a mut self) -> View<'a> {
let ready = self.input.ready_state();
View {
position: &self.position,
play_time: &self.play_time,
volume: &mut self.volume,
meta: self.input.metadata(),
ready,
playing: &mut self.playing,
loops: &mut self.loops,
}
}
pub(crate) fn process_commands(&mut self, index: usize, ic: &Interconnect) -> Action {
// Note: disconnection and an empty channel are both valid,
// and should allow the audio object to keep running as intended.
// We also need to export a target seek point to the mixer, if known.
let mut action = Action::default();
// Note that interconnect failures are not currently errors.
// In correct operation, the event thread should never panic,
// but it receiving status updates is secondary do actually
// doing the work.
while let Ok(cmd) = self.commands.try_recv() {
match cmd {
TrackCommand::Play => {
self.playing.change_to(PlayMode::Play);
drop(ic.events.send(EventMessage::ChangeState(
index,
TrackStateChange::Mode(self.playing.clone()),
)));
},
TrackCommand::Pause => {
self.playing.change_to(PlayMode::Pause);
drop(ic.events.send(EventMessage::ChangeState(
index,
TrackStateChange::Mode(self.playing.clone()),
)));
},
TrackCommand::Stop => {
self.playing.change_to(PlayMode::Stop);
drop(ic.events.send(EventMessage::ChangeState(
index,
TrackStateChange::Mode(self.playing.clone()),
)));
},
TrackCommand::Volume(vol) => {
self.volume = vol;
drop(ic.events.send(EventMessage::ChangeState(
index,
TrackStateChange::Volume(self.volume),
)));
},
TrackCommand::Seek(req) => action.seek_point = Some(req),
TrackCommand::AddEvent(evt) => {
drop(ic.events.send(EventMessage::AddTrackEvent(index, evt)));
},
TrackCommand::Do(func) => {
if let Some(indiv_action) = func(self.view()) {
action.combine(indiv_action);
}
drop(ic.events.send(EventMessage::ChangeState(
index,
TrackStateChange::Total(self.state()),
)));
},
TrackCommand::Request(tx) => {
drop(tx.send(self.state()));
},
TrackCommand::Loop(loops) => {
self.loops = loops;
drop(ic.events.send(EventMessage::ChangeState(
index,
TrackStateChange::Loops(self.loops, true),
)));
},
TrackCommand::MakePlayable(callback) => action.make_playable = Some(callback),
}
}
action
}
pub(crate) fn do_loop(&mut self) -> bool {
match &mut self.loops {
LoopState::Infinite => true,
LoopState::Finite(n) =>
if let Some(new_n) = nonmax::NonMaxU32::new(n.get().wrapping_sub(1)) {
*n = new_n;
true
} else {
false
},
}
}
/// Steps playback location forward by one frame.
pub(crate) fn step_frame(&mut self) {
self.position += TIMESTEP_LENGTH;
self.play_time += TIMESTEP_LENGTH;
}
pub(crate) fn should_check_input(&self) -> bool {
self.playing.is_playing() || matches!(self.input, InputState::Preparing(_))
}
pub(crate) fn end(&mut self) -> &mut Self {
self.playing.change_to(PlayMode::End);
self
}
/// Readies the requested input state.
///
/// Returns the usable version of the audio if available, and whether the track should be deleted.
pub(crate) fn get_or_ready_input(
&'a mut self,
id: usize,
interconnect: &Interconnect,
pool: &BlockyTaskPool,
config: &Arc<Config>,
prevent_events: bool,
) -> StdResult<(&'a mut Parsed, &'a mut DecodeState), InputReadyingError> {
let input = &mut self.input;
let mix_state = &mut self.mix_state;
let (out, queued_seek) = match input {
InputState::NotReady(_) => {
let (tx, rx) = flume::bounded(1);
let mut state = InputState::Preparing(PreparingInfo {
time: Instant::now(),
queued_seek: None,
callback: rx,
});
std::mem::swap(&mut state, input);
match state {
InputState::NotReady(a @ Input::Lazy(_)) => {
pool.create(tx, a, None, config.clone());
},
InputState::NotReady(Input::Live(audio, rec)) => {
pool.parse(config.clone(), tx, audio, rec, None);
},
_ => unreachable!(),
}
if !prevent_events {
drop(interconnect.events.send(EventMessage::ChangeState(
id,
TrackStateChange::Ready(ReadyState::Preparing),
)));
}
(Err(InputReadyingError::Waiting), None)
},
InputState::Preparing(info) => {
let queued_seek = info.queued_seek.take();
let orig_out = match info.callback.try_recv() {
Ok(MixerInputResultMessage::Built(parsed, rec)) => {
*input = InputState::Ready(parsed, rec);
mix_state.reset();
// possible TODO: set position to the true track position here?
// ISSUE: need to get next_packet to see its `ts`, but inner_pos==0
// will trigger next packet to be taken at mix time.
if !prevent_events {
drop(interconnect.events.send(EventMessage::ChangeState(
id,
TrackStateChange::Ready(ReadyState::Playable),
)));
}
self.callbacks.playable();
if let InputState::Ready(ref mut parsed, _) = input {
Ok(parsed)
} else {
unreachable!()
}
},
Ok(MixerInputResultMessage::Seek(parsed, rec, seek_res)) => {
match seek_res {
Ok(pos) => {
if let Some(time_base) = parsed.decoder.codec_params().time_base {
// Update track's position to match the actual timestamp the
// seek landed at.
let new_time = time_base.calc_time(pos.actual_ts);
let time_in_float = new_time.seconds as f64 + new_time.frac;
self.position =
std::time::Duration::from_secs_f64(time_in_float);
self.callbacks.seeked(self.position);
self.callbacks.playable();
if !prevent_events {
drop(interconnect.events.send(EventMessage::ChangeState(
id,
TrackStateChange::Position(self.position),
)));
drop(interconnect.events.send(EventMessage::ChangeState(
id,
TrackStateChange::Ready(ReadyState::Playable),
)));
}
// Our decoder state etc. must be reset.
// (Symphonia decoder state reset in the thread pool during
// the operation.)
mix_state.reset();
*input = InputState::Ready(parsed, rec);
if let InputState::Ready(ref mut parsed, _) = input {
Ok(parsed)
} else {
unreachable!()
}
} else {
Err(InputReadyingError::Seeking(
SymphError::Unsupported("Track had no recorded time base.")
.into(),
))
}
},
Err(e) => Err(InputReadyingError::Seeking(e)),
}
},
Ok(MixerInputResultMessage::CreateErr(e)) =>
Err(InputReadyingError::Creation(e)),
Ok(MixerInputResultMessage::ParseErr(e)) => Err(InputReadyingError::Parsing(e)),
Err(TryRecvError::Disconnected) => Err(InputReadyingError::Dropped),
Err(TryRecvError::Empty) => Err(InputReadyingError::Waiting),
};
let orig_out = orig_out.map(|a| (a, mix_state));
if let Err(ref e) = orig_out {
if let Some(e) = e.as_user() {
self.callbacks.readying_error(e);
}
}
(orig_out, queued_seek)
},
InputState::Ready(ref mut parsed, _) => (Ok((parsed, mix_state)), None),
};
match (out, queued_seek) {
(Ok(_), Some(request)) => Err(InputReadyingError::NeedsSeek(request)),
(a, _) => a,
}
}
pub(crate) fn seek(
&mut self,
id: usize,
request: SeekRequest,
interconnect: &Interconnect,
pool: &BlockyTaskPool,
config: &Arc<Config>,
prevent_events: bool,
) {
if let InputState::Preparing(p) = &mut self.input {
p.queued_seek = Some(request);
return;
}
// might be a little topsy turvy: rethink me.
let SeekRequest { time, callback } = request;
self.callbacks.seek = Some(callback);
if !prevent_events {
drop(interconnect.events.send(EventMessage::ChangeState(
id,
TrackStateChange::Ready(ReadyState::Preparing),
)));
}
let backseek_needed = time < self.position;
let time = Time::from(time.as_secs_f64());
let mut ts = SeekTo::Time {
time,
track_id: None,
};
let (tx, rx) = flume::bounded(1);
let state = std::mem::replace(
&mut self.input,
InputState::Preparing(PreparingInfo {
time: Instant::now(),
callback: rx,
queued_seek: None,
}),
);
match state {
InputState::Ready(p, r) => {
if let SeekTo::Time { time: _, track_id } = &mut ts {
*track_id = Some(p.track_id);
}
pool.seek(tx, p, r, ts, backseek_needed, config.clone());
},
InputState::NotReady(lazy) => pool.create(tx, lazy, Some(ts), config.clone()),
InputState::Preparing(_) => unreachable!(), // Covered above.
}
}
}
#[derive(Debug, Default)]
pub struct Callbacks {
pub seek: Option<Sender<StdResult<Duration, PlayError>>>,
pub make_playable: Option<Sender<StdResult<(), PlayError>>>,
}
impl Callbacks {
fn readying_error(&mut self, err: PlayError) {
if let Some(callback) = self.seek.take() {
drop(callback.send(Err(err.clone())));
}
if let Some(callback) = self.make_playable.take() {
drop(callback.send(Err(err)));
}
}
fn playable(&mut self) {
if let Some(callback) = self.make_playable.take() {
drop(callback.send(Ok(())));
}
}
fn seeked(&mut self, time: Duration) {
if let Some(callback) = self.seek.take() {
drop(callback.send(Ok(time)));
}
}
}

View File

@@ -0,0 +1,20 @@
use symphonia_core::{formats::SeekTo, units::Time};
// SeekTo lacks Copy and Clone... somehow.
pub fn copy_seek_to(pos: &SeekTo) -> SeekTo {
match *pos {
SeekTo::Time { time, track_id } => SeekTo::Time { time, track_id },
SeekTo::TimeStamp { ts, track_id } => SeekTo::TimeStamp { ts, track_id },
}
}
pub fn seek_to_is_zero(pos: &SeekTo) -> bool {
match *pos {
SeekTo::Time { time, .. } =>
time == Time {
seconds: 0,
frac: 0.0,
},
SeekTo::TimeStamp { ts, .. } => ts == 0,
}
}

View File

@@ -0,0 +1,351 @@
#![allow(missing_docs)]
pub(crate) mod disposal;
pub mod error;
mod events;
pub mod message;
pub mod mixer;
#[cfg(feature = "receive")]
pub(crate) mod udp_rx;
pub(crate) mod ws;
use super::connection::{error::Error as ConnectionError, Connection};
use crate::{
events::{
context_data::{DisconnectKind, DisconnectReason},
internal_data::{InternalConnect, InternalDisconnect},
CoreContext,
},
Config,
ConnectionInfo,
FloatDuration,
};
use flume::{Receiver, Sender};
use message::*;
use tokio::{spawn, time::sleep as tsleep};
use tracing::{debug, instrument, trace};
pub(crate) fn start(config: Config, rx: Receiver<CoreMessage>, tx: Sender<CoreMessage>) {
spawn(async move {
trace!("Driver started.");
runner(config, rx, tx).await;
trace!("Driver finished.");
});
}
fn start_internals(core: Sender<CoreMessage>, config: &Config) -> Interconnect {
let (evt_tx, evt_rx) = flume::unbounded();
let (mix_tx, mix_rx) = flume::unbounded();
spawn(async move {
trace!("Event processor started.");
events::runner(evt_rx).await;
trace!("Event processor finished.");
});
let ic = Interconnect {
core,
events: evt_tx,
mixer: mix_tx,
};
config.get_scheduler().new_mixer(config, ic.clone(), mix_rx);
ic
}
#[instrument(skip(rx, tx))]
async fn runner(mut config: Config, rx: Receiver<CoreMessage>, tx: Sender<CoreMessage>) {
let mut next_config: Option<Config> = None;
let mut connection: Option<Connection> = None;
let mut interconnect = start_internals(tx, &config);
let mut retrying = None;
let mut attempt_idx = 0;
while let Ok(msg) = rx.recv_async().await {
match msg {
CoreMessage::ConnectWithResult(info, tx) => {
config = if let Some(new_config) = next_config.take() {
drop(
interconnect
.mixer
.send(MixerMessage::SetConfig(new_config.clone())),
);
new_config
} else {
config
};
if connection.as_ref().is_none_or(|conn| conn.info != info) {
// Only *actually* reconnect if the conn info changed, or we don't have an
// active connection.
// This allows the gateway component to keep sending join requests independent
// of driver failures.
connection = ConnectionRetryData::connect(tx, info, &mut attempt_idx)
.attempt(&mut retrying, &interconnect, &config)
.await;
} else {
// No reconnection was attempted as there's a valid, identical connection;
// tell the outside listener that the operation was a success.
drop(tx.send(Ok(())));
}
},
CoreMessage::RetryConnect(retry_idx) => {
debug!("Retrying idx: {} (vs. {})", retry_idx, attempt_idx);
if retry_idx == attempt_idx {
if let Some(progress) = retrying.take() {
connection = progress
.attempt(&mut retrying, &interconnect, &config)
.await;
}
}
},
CoreMessage::Disconnect => {
let last_conn = connection.take();
drop(interconnect.mixer.send(MixerMessage::DropConn));
drop(interconnect.mixer.send(MixerMessage::RebuildEncoder));
if let Some(conn) = last_conn {
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::DriverDisconnect(InternalDisconnect {
kind: DisconnectKind::Runtime,
reason: Some(DisconnectReason::Requested),
info: conn.info.clone(),
}),
)));
}
},
CoreMessage::SignalWsClosure(ws_idx, ws_info, mut reason) => {
// if idx is not a match, quash reason
// (i.e., prevent users from mistakenly trying to reconnect for an *old* dead conn).
// if it *is* a match, the conn needs to die!
// (as the WS channel has truly given up the ghost).
let conn = if ws_idx == attempt_idx {
drop(interconnect.mixer.send(MixerMessage::DropConn));
drop(interconnect.mixer.send(MixerMessage::RebuildEncoder));
connection.take()
} else {
reason = None;
None
};
// Conn may have been unset earlier (i.e., in a deliberate disconnect).
// If so, do not repropagate/repeat the disconnect event.
if conn.is_some() {
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::DriverDisconnect(InternalDisconnect {
kind: DisconnectKind::Runtime,
reason,
info: ws_info,
}),
)));
}
},
CoreMessage::SetTrack(s) => {
drop(interconnect.mixer.send(MixerMessage::SetTrack(s)));
},
CoreMessage::AddTrack(s) => {
drop(interconnect.mixer.send(MixerMessage::AddTrack(s)));
},
CoreMessage::SetBitrate(b) => {
drop(interconnect.mixer.send(MixerMessage::SetBitrate(b)));
},
CoreMessage::SetConfig(mut new_config) => {
next_config = Some(new_config.clone());
new_config.make_safe(&config, connection.is_some());
drop(interconnect.mixer.send(MixerMessage::SetConfig(new_config)));
},
CoreMessage::AddEvent(evt) => {
drop(interconnect.events.send(EventMessage::AddGlobalEvent(evt)));
},
CoreMessage::RemoveGlobalEvents => {
drop(interconnect.events.send(EventMessage::RemoveGlobalEvents));
},
CoreMessage::Mute(m) => {
drop(interconnect.mixer.send(MixerMessage::SetMute(m)));
},
CoreMessage::Reconnect => {
if let Some(mut conn) = connection.take() {
// try once: if interconnect, try again.
// if still issue, full connect.
let info = conn.info.clone();
let full_connect = match conn.reconnect(&config).await {
Ok(()) => {
connection = Some(conn);
false
},
Err(ConnectionError::InterconnectFailure(_)) => {
interconnect.restart_volatile_internals();
match conn.reconnect(&config).await {
Ok(()) => {
connection = Some(conn);
false
},
_ => true,
}
},
_ => true,
};
if full_connect {
connection = ConnectionRetryData::reconnect(info, &mut attempt_idx)
.attempt(&mut retrying, &interconnect, &config)
.await;
} else if let Some(ref connection) = &connection {
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::DriverReconnect(InternalConnect {
info: connection.info.clone(),
ssrc: connection.ssrc,
}),
)));
}
}
},
CoreMessage::FullReconnect =>
if let Some(conn) = connection.take() {
let info = conn.info.clone();
connection = ConnectionRetryData::reconnect(info, &mut attempt_idx)
.attempt(&mut retrying, &interconnect, &config)
.await;
},
CoreMessage::RebuildInterconnect => {
interconnect.restart_volatile_internals();
},
CoreMessage::Poison => break,
}
}
trace!("Main thread exited");
interconnect.poison_all();
}
struct ConnectionRetryData {
flavour: ConnectionFlavour,
attempts: u8,
last_wait: Option<FloatDuration>,
info: ConnectionInfo,
idx: usize,
}
impl ConnectionRetryData {
fn connect(
tx: Sender<Result<(), ConnectionError>>,
info: ConnectionInfo,
idx_src: &mut usize,
) -> Self {
Self::base(ConnectionFlavour::Connect(tx), info, idx_src)
}
fn reconnect(info: ConnectionInfo, idx_src: &mut usize) -> Self {
Self::base(ConnectionFlavour::Reconnect, info, idx_src)
}
fn base(flavour: ConnectionFlavour, info: ConnectionInfo, idx_src: &mut usize) -> Self {
*idx_src = idx_src.wrapping_add(1);
Self {
flavour,
attempts: 0,
last_wait: None,
info,
idx: *idx_src,
}
}
async fn attempt(
mut self,
attempt_slot: &mut Option<Self>,
interconnect: &Interconnect,
config: &Config,
) -> Option<Connection> {
match Connection::new(self.info.clone(), interconnect, config, self.idx).await {
Ok(connection) => {
match self.flavour {
ConnectionFlavour::Connect(tx) => {
// Other side may not be listening: this is fine.
drop(tx.send(Ok(())));
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::DriverConnect(InternalConnect {
info: connection.info.clone(),
ssrc: connection.ssrc,
}),
)));
},
ConnectionFlavour::Reconnect => {
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::DriverReconnect(InternalConnect {
info: connection.info.clone(),
ssrc: connection.ssrc,
}),
)));
},
}
Some(connection)
},
Err(why) => {
debug!("Failed to connect for {:?}: {}", self.info.guild_id, why);
if let Some(t) = config.driver_retry.retry_in(self.last_wait, self.attempts) {
let remote_ic = interconnect.clone();
let idx = self.idx;
spawn(async move {
tsleep(t.into()).await;
drop(remote_ic.core.send(CoreMessage::RetryConnect(idx)));
});
self.attempts += 1;
self.last_wait = Some(t);
debug!(
"Retrying connection for {:?} in {}s ({}/{:?})",
self.info.guild_id,
t.as_secs_f32(),
self.attempts,
config.driver_retry.retry_limit
);
*attempt_slot = Some(self);
} else {
let reason = Some(DisconnectReason::from(&why));
match self.flavour {
ConnectionFlavour::Connect(tx) => {
// See above.
drop(tx.send(Err(why)));
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::DriverDisconnect(InternalDisconnect {
kind: DisconnectKind::Connect,
reason,
info: self.info,
}),
)));
},
ConnectionFlavour::Reconnect => {
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::DriverDisconnect(InternalDisconnect {
kind: DisconnectKind::Reconnect,
reason,
info: self.info,
}),
)));
},
}
}
None
},
}
}
}
enum ConnectionFlavour {
Connect(Sender<Result<(), ConnectionError>>),
Reconnect,
}

View File

@@ -0,0 +1,42 @@
use crate::constants::STEREO_FRAME_SIZE;
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum PacketDecodeSize {
/// Minimum frame size on Discord.
TwentyMillis,
/// Hybrid packet, sent by Firefox web client.
///
/// Likely 20ms frame + 10ms frame.
ThirtyMillis,
/// Next largest frame size.
FortyMillis,
/// Maximum Opus frame size.
SixtyMillis,
/// Maximum Opus packet size: 120ms.
Max,
}
impl PacketDecodeSize {
pub fn bump_up(self) -> Self {
match self {
Self::TwentyMillis => Self::ThirtyMillis,
Self::ThirtyMillis => Self::FortyMillis,
Self::FortyMillis => Self::SixtyMillis,
Self::SixtyMillis | Self::Max => Self::Max,
}
}
pub fn can_bump_up(self) -> bool {
self != Self::Max
}
pub fn len(self) -> usize {
match self {
Self::TwentyMillis => STEREO_FRAME_SIZE,
Self::ThirtyMillis => (STEREO_FRAME_SIZE / 2) * 3,
Self::FortyMillis => 2 * STEREO_FRAME_SIZE,
Self::SixtyMillis => 3 * STEREO_FRAME_SIZE,
Self::Max => 6 * STEREO_FRAME_SIZE,
}
}
}

View File

@@ -0,0 +1,114 @@
//! Jarvis patch: RTP packets whose SSRC Discord has not named yet are held for a moment.
//!
//! Discord names a speaker's SSRC in a Speaking op over the voice websocket the first time they speak,
//! and that op arrives after their first packets over UDP. Unpatched, songbird dropped DAVE packets
//! of an unnamed SSRC (it needs the user to decrypt them), which cut the first ~100 ms off the first
//! thing each person said after a join ("자비스" heard as "아비"). Held packets are processed in
//! arrival order before the first packet after the name arrives.
use bytes::BytesMut;
use std::collections::{HashMap, VecDeque};
use std::time::Duration;
use tokio::time::Instant;
/// A packet older than this when its SSRC is named is dropped.
pub(crate) const HOLD: Duration = Duration::from_secs(1);
/// At most this many packets per SSRC: one second of 20 ms packets.
pub(crate) const MAX_PACKETS: usize = 50;
/// At most this many unnamed SSRCs at once.
pub(crate) const MAX_SSRCS: usize = 32;
#[derive(Default)]
pub(crate) struct Held {
by_ssrc: HashMap<u32, VecDeque<(Instant, BytesMut)>>,
}
impl Held {
pub(crate) fn hold(&mut self, ssrc: u32, at: Instant, packet: BytesMut) {
if !self.by_ssrc.contains_key(&ssrc) && self.by_ssrc.len() >= MAX_SSRCS {
return;
}
let queue = self.by_ssrc.entry(ssrc).or_default();
while queue.front().is_some_and(|(t, _)| at.duration_since(*t) > HOLD) || queue.len() >= MAX_PACKETS {
queue.pop_front();
}
queue.push_back((at, packet));
}
/// The packets held for `ssrc` that are at most `HOLD` old, oldest first. Forgets all of them.
pub(crate) fn take(&mut self, ssrc: u32, now: Instant) -> Vec<BytesMut> {
self.by_ssrc
.remove(&ssrc)
.map(|q| q.into_iter().filter(|(t, _)| now.duration_since(*t) <= HOLD).map(|(_, p)| p).collect())
.unwrap_or_default()
}
/// Forgets every SSRC whose newest packet is older than `HOLD`.
pub(crate) fn prune(&mut self, now: Instant) {
self.by_ssrc.retain(|_, q| q.back().is_some_and(|(t, _)| now.duration_since(*t) <= HOLD));
}
}
#[cfg(test)]
mod tests {
use super::*;
fn p(n: u8) -> BytesMut {
BytesMut::from(&[n][..])
}
#[test]
fn packets_held_before_the_name_come_back_in_arrival_order() {
let mut h = Held::default();
let t = Instant::now();
for n in 0..3 {
h.hold(7, t + Duration::from_millis(20 * u64::from(n)), p(n));
}
h.hold(8, t, p(9));
assert_eq!(h.take(7, t + Duration::from_millis(100)), vec![p(0), p(1), p(2)]);
assert!(h.take(7, t + Duration::from_millis(100)).is_empty(), "taken once");
assert_eq!(h.take(8, t), vec![p(9)], "other speakers are kept apart");
}
#[test]
fn packets_older_than_the_hold_are_dropped() {
let mut h = Held::default();
let t = Instant::now();
h.hold(7, t, p(0));
h.hold(7, t + Duration::from_millis(900), p(1));
assert_eq!(h.take(7, t + Duration::from_millis(1500)), vec![p(1)]);
h.hold(7, t, p(0));
h.hold(7, t + HOLD + Duration::from_millis(20), p(1));
assert_eq!(h.take(7, t + HOLD + Duration::from_millis(20)), vec![p(1)], "dropped as newer ones come");
}
#[test]
fn at_most_a_second_of_packets_per_speaker_and_a_bounded_number_of_speakers() {
let mut h = Held::default();
let t = Instant::now();
for n in 0..(MAX_PACKETS as u8 + 5) {
h.hold(7, t, p(n));
}
let kept = h.take(7, t);
assert_eq!(kept.len(), MAX_PACKETS);
assert_eq!(kept[0], p(5), "the newest are kept");
for ssrc in 0..(MAX_SSRCS as u32 + 3) {
h.hold(100 + ssrc, t, p(1));
}
assert!(h.take(100 + MAX_SSRCS as u32, t).is_empty());
assert_eq!(h.take(100, t), vec![p(1)]);
}
#[test]
fn prune_forgets_speakers_that_stopped_sending() {
let mut h = Held::default();
let t = Instant::now();
h.hold(7, t, p(0));
h.hold(8, t + Duration::from_millis(1500), p(1));
h.prune(t + Duration::from_millis(1600));
assert!(h.take(7, t).is_empty());
assert_eq!(h.take(8, t + Duration::from_millis(1600)), vec![p(1)]);
}
}

View File

@@ -0,0 +1,442 @@
mod decode_sizes;
mod held;
mod playout_buffer;
mod ssrc_state;
use self::{decode_sizes::*, held::Held, playout_buffer::*, ssrc_state::*};
use super::message::*;
use crate::driver::{CryptoMode, DecodeMode};
use crate::{
constants::*,
driver::crypto::Cipher,
events::{context_data::VoiceTick, internal_data::*, CoreContext},
Config,
};
use bytes::BytesMut;
use discortp::{
demux::{self, DemuxedMut},
rtp::RtpPacket,
};
use discortp::{MutablePacket, Packet};
use flume::Receiver;
use std::sync::{
atomic::{AtomicU16, Ordering},
RwLock,
};
use std::{
collections::{HashMap, HashSet},
num::Wrapping,
sync::Arc,
time::Duration,
};
use tokio::{net::UdpSocket, select, time::Instant};
use tracing::{error, instrument, trace, warn};
type RtpSequence = Wrapping<u16>;
type RtpTimestamp = Wrapping<u32>;
type RtpSsrc = u32;
struct UdpRx {
cipher: Cipher,
crypto_mode: CryptoMode,
decoder_map: HashMap<RtpSsrc, SsrcState>,
config: Config,
rx: Receiver<UdpRxMessage>,
ssrc_signalling: Arc<SsrcTracker>,
udp_socket: UdpSocket,
dave_session: Arc<RwLock<Option<davey::DaveSession>>>,
dave_protocol_version: Arc<AtomicU16>,
/// Jarvis patch: packets of SSRCs not named yet (see held.rs).
held: Held,
}
impl UdpRx {
#[instrument(skip(self))]
async fn run(&mut self, interconnect: &mut Interconnect) {
let mut cleanup_time = Instant::now();
let mut playout_time = Instant::now() + TIMESTEP_LENGTH;
let mut byte_dest: Option<BytesMut> = None;
loop {
if byte_dest.is_none() {
byte_dest = Some(BytesMut::zeroed(VOICE_PACKET_MAX));
}
select! {
Ok((len, _addr)) = self.udp_socket.recv_from(byte_dest.as_mut().unwrap()) => {
let mut pkt = byte_dest.take().unwrap();
pkt.truncate(len);
self.process_udp_message(interconnect, pkt).await;
},
msg = self.rx.recv_async() => {
match msg {
Ok(UdpRxMessage::ReplaceInterconnect(i)) => {
*interconnect = i;
},
Ok(UdpRxMessage::SetConfig(new_config)) => {
if let DecodeMode::Decode(old_config) = &mut self.config.decode_mode {
if *old_config != new_config {
*old_config = new_config;
self.decoder_map.values_mut().for_each(|v| v.reconfigure_decoder(new_config));
}
}
},
Err(flume::RecvError::Disconnected) => break,
}
},
() = tokio::time::sleep_until(playout_time) => {
let mut tick = VoiceTick {
speaking: HashMap::new(),
silent: HashSet::new(),
};
for (ssrc, state) in &mut self.decoder_map {
match state.get_voice_tick(&self.config) {
Ok(Some(data)) => {
tick.speaking.insert(*ssrc, data);
},
Ok(None) => {
if !state.disconnected {
tick.silent.insert(*ssrc);
}
},
Err(e) => {
warn!("Decode error for SSRC {ssrc}: {e:?}");
tick.silent.insert(*ssrc);
},
}
}
playout_time += TIMESTEP_LENGTH;
drop(interconnect.events.send(EventMessage::FireCoreEvent(CoreContext::VoiceTick(tick))));
},
() = tokio::time::sleep_until(cleanup_time) => {
// periodic cleanup.
let now = Instant::now();
// check ssrc map to see if the WS task has informed us of any disconnects.
loop {
// This is structured in an odd way to prevent deadlocks.
// while-let seemed to keep the dashmap iter() alive for block scope, rather than
// just the initialiser.
let id = {
if let Some(id) = self.ssrc_signalling.disconnected_users.iter().next().map(|v| *v.key()) {
id
} else {
break;
}
};
_ = self.ssrc_signalling.disconnected_users.remove(&id);
if let Some((_, ssrc)) = self.ssrc_signalling.user_ssrc_map.remove(&id) {
let _ = self.ssrc_signalling.ssrc_user_map.remove(&ssrc);
if let Some(state) = self.decoder_map.get_mut(&ssrc) {
// don't cleanup immediately: leave for later cycle
// this is key with reorder/jitter buffers where we may
// still need to decode post disconnect for ~0.2s.
state.prune_time = now + Duration::from_secs(1);
state.disconnected = true;
}
}
}
// now remove all dead ssrcs.
self.decoder_map.retain(|_, v| v.prune_time > now);
self.held.prune(now);
cleanup_time = now + Duration::from_secs(5);
},
}
}
}
/// Jarvis patch: an RTP packet of an SSRC that Discord has not named yet is held, and the held
/// ones are processed before the first packet after the name came (see held.rs).
async fn process_udp_message(&mut self, interconnect: &Interconnect, packet: BytesMut) {
if let demux::Demuxed::Rtp(rtp) = demux::demux(&packet) {
let ssrc = rtp.get_ssrc();
let now = Instant::now();
if !self.ssrc_signalling.ssrc_user_map.contains_key(&ssrc) {
self.held.hold(ssrc, now, packet);
return;
}
for held in self.held.take(ssrc, now) {
self.process_packet(interconnect, held).await;
}
}
self.process_packet(interconnect, packet).await;
}
async fn process_packet(&mut self, interconnect: &Interconnect, mut packet: BytesMut) {
// NOTE: errors here (and in general for UDP) are not fatal to the connection.
// Panics should be avoided due to adversarial nature of rx'd packets,
// but correct handling should not prompt a reconnect.
//
// For simplicity, if the event task fails then we nominate the mixing thread
// to rebuild their context etc. (hence, the `let _ =` statements.), as it will
// try to make contact every 20ms.
let crypto_mode = self.crypto_mode;
match demux::demux_mut(packet.as_mut()) {
DemuxedMut::Rtp(mut rtp) => {
if !rtp_valid(&rtp.to_immutable()) {
error!("Illegal RTP message received.");
return;
}
let mut packet_data = if self.config.decode_mode.should_decrypt() {
let out = self.cipher.decrypt_rtp_in_place(&mut rtp).map(|(s, t)| {
if rtp.get_padding() != 0 {
let payload = rtp.payload();
let payload_length = payload.len();
let padding_count = payload[payload_length - t - 1] as usize;
(s, t + padding_count, true)
} else {
(s, t, true)
}
});
if let Err(ref e) = out {
warn!("RTP decryption failed: {:?}", e);
}
out.ok()
} else {
None
};
if let Some((rtp_body_start, rtp_body_tail, decrypted)) = packet_data {
let ssrc = rtp.get_ssrc();
let payload = rtp.payload_mut();
let payload_length = payload.len();
let body = &mut payload[rtp_body_start..payload_length - rtp_body_tail];
let body_length = body.len();
// If the packet is transport-decrypted, DAVE is enabled, and the packet
// looks encrypted: https://daveprotocol.com/#protocol-frame-check
// The packet must be at least 11 bytes consisting of:
// - 8 byte truncated AES128-GCM authentication tag
// - 1 byte protocol supplemental data size
// - 2 byte magic marker
if decrypted
&& self.dave_protocol_version.load(Ordering::Relaxed) != 0
&& body_length >= 11
&& body[body_length - DAVE_MAGIC_MARKER.len()..body_length]
== DAVE_MAGIC_MARKER
{
// Silently drop encrypted packets if it's not possible to decrypt them for
// one reason or another; otherwise there'd be error logs for trying to decode
// Opus
let Some(user_id) = self.ssrc_signalling.ssrc_user_map.get(&ssrc) else {
return;
};
let Some(ref mut dave_session) = *self.dave_session.write().unwrap() else {
return;
};
if !dave_session.is_ready() {
return;
}
let result = dave_session.decrypt(user_id.0, davey::MediaType::AUDIO, body);
match result {
Ok(decrypted_body) => {
packet_data = Some((
rtp_body_start,
rtp_body_tail + (body.len() - decrypted_body.len()),
decrypted,
));
body[..decrypted_body.len()].copy_from_slice(&decrypted_body);
},
Err(davey::errors::DecryptError::NoDecryptorForUser)
| Err(davey::errors::DecryptError::DecryptionFailed(
davey::errors::DecryptorDecryptError::NoValidCryptorFound {
..
},
)) => {
// Silently drop encrypted packets for users whose ratchets are not configured yet.
return;
},
Err(e) => {
error!(error = ?e, "DAVE decryption failed");
return;
},
}
}
}
let rtp = rtp.to_immutable();
let (rtp_body_start, rtp_body_tail, decrypted) = packet_data.unwrap_or_else(|| {
(
crypto_mode.payload_prefix_len(),
crypto_mode.payload_suffix_len(),
false,
)
});
let entry = self
.decoder_map
.entry(rtp.get_ssrc())
.or_insert_with(|| SsrcState::new(&rtp, crypto_mode, &self.config));
// Only do this on RTP, rather than RTCP -- this pins decoder state liveness
// to *speech* rather than just presence.
entry.refresh_timer(self.config.decode_state_timeout.into());
let store_pkt = StoredPacket {
packet: packet.freeze(),
decrypted,
};
let packet = store_pkt.packet.clone();
entry.store_packet(store_pkt, &self.config);
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::RtpPacket(InternalRtpPacket {
packet,
payload_offset: rtp_body_start,
payload_end_pad: rtp_body_tail,
}),
)));
},
DemuxedMut::Rtcp(mut rtcp) => {
let packet_data = if self.config.decode_mode.should_decrypt() {
let out = self.cipher.decrypt_rtcp_in_place(&mut rtcp);
if let Err(ref e) = out {
warn!("RTCP decryption failed: {:?}", e);
}
out.ok()
} else {
None
};
let (start, tail) = packet_data.unwrap_or_else(|| {
(
crypto_mode.payload_prefix_len(),
crypto_mode.payload_suffix_len(),
)
});
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::RtcpPacket(InternalRtcpPacket {
packet: packet.freeze(),
payload_offset: start,
payload_end_pad: tail,
}),
)));
},
DemuxedMut::FailedParse(t) => {
warn!("Failed to parse message of type {:?}.", t);
},
DemuxedMut::TooSmall => {
warn!("Illegal UDP packet from voice server.");
},
}
}
}
#[instrument(skip(interconnect, rx, cipher))]
pub(crate) async fn runner(
mut interconnect: Interconnect,
rx: Receiver<UdpRxMessage>,
cipher: Cipher,
crypto_mode: CryptoMode,
config: Config,
udp_socket: UdpSocket,
ssrc_signalling: Arc<SsrcTracker>,
dave_session: Arc<RwLock<Option<davey::DaveSession>>>,
dave_protocol_version: Arc<AtomicU16>,
) {
trace!("UDP receive handle started.");
let mut state = UdpRx {
cipher,
crypto_mode,
decoder_map: HashMap::new(),
config,
rx,
ssrc_signalling,
udp_socket,
dave_session,
dave_protocol_version,
held: Held::default(),
};
state.run(&mut interconnect).await;
trace!("UDP receive handle stopped.");
}
#[inline]
fn rtp_valid(packet: &RtpPacket<'_>) -> bool {
packet.get_version() == RTP_VERSION && packet.get_payload_type() == RTP_PROFILE_TYPE
}
#[cfg(test)]
mod jarvis_held_tests {
//! Jarvis patch (held.rs): the first packets of a speaker are not lost to a late Speaking op.
use super::*;
use crate::driver::DecodeMode;
use discortp::rtp::MutableRtpPacket;
use serenity_voice_model::id::UserId;
fn rtp(seq: u16, ssrc: u32) -> BytesMut {
let mut buf = BytesMut::zeroed(MutableRtpPacket::minimum_packet_size() + 3);
let mut p = MutableRtpPacket::new(&mut buf[..]).unwrap();
p.set_version(RTP_VERSION);
p.set_payload_type(RTP_PROFILE_TYPE);
p.set_sequence(seq.into());
p.set_ssrc(ssrc);
buf
}
fn sequences(events: &Receiver<EventMessage>) -> Vec<u16> {
events
.try_iter()
.filter_map(|e| match e {
EventMessage::FireCoreEvent(CoreContext::RtpPacket(p)) =>
Some(RtpPacket::new(&p.packet).unwrap().get_sequence().0 .0),
_ => None,
})
.collect()
}
#[tokio::test]
async fn packets_before_the_speaking_op_reach_handlers_once_it_names_the_ssrc() {
let (core, _core_rx) = flume::unbounded();
let (events, events_rx) = flume::unbounded();
let (mixer, _mixer_rx) = flume::unbounded();
let ic = Interconnect { core, events, mixer };
let mode = CryptoMode::Aes256Gcm;
let mut rx = UdpRx {
cipher: mode.cipher_from_key(&[0u8; 32]).unwrap(),
crypto_mode: mode,
decoder_map: HashMap::new(),
config: Config::default().decode_mode(DecodeMode::Pass),
rx: flume::unbounded().1,
ssrc_signalling: Arc::new(SsrcTracker::default()),
udp_socket: UdpSocket::bind("127.0.0.1:0").await.unwrap(),
dave_session: Arc::new(RwLock::new(None)),
dave_protocol_version: Arc::new(AtomicU16::new(0)),
held: Held::default(),
};
rx.process_udp_message(&ic, rtp(1, 42)).await;
rx.process_udp_message(&ic, rtp(2, 42)).await;
assert!(sequences(&events_rx).is_empty(), "held until the SSRC is named");
rx.ssrc_signalling.ssrc_user_map.insert(42, UserId(7));
rx.process_udp_message(&ic, rtp(3, 42)).await;
assert_eq!(sequences(&events_rx), vec![1, 2, 3]);
rx.process_udp_message(&ic, rtp(4, 42)).await;
assert_eq!(sequences(&events_rx), vec![4], "a held packet is processed once");
}
}

View File

@@ -0,0 +1,204 @@
use super::*;
use bytes::Bytes;
use std::collections::VecDeque;
use tracing::trace;
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct StoredPacket {
pub packet: Bytes,
// We need to store this as it's possible that a user can change config modes.
pub decrypted: bool,
}
/// Determines whether an SSRC's packets should be decoded.
///
/// Playout requires us to keep an almost constant delay, to do so we build
/// a user's packet buffer up to the required length ([`Config::playout_buffer_length`])
/// ([`Self::Fill`]) and then emit packets on each tick ([`Self::Drain`]).
///
/// This gets a bit harder to reason about when users stop speaking. If a speech gap
/// lasts longer than the playout buffer, then we can simply swap from `Drain` -> `Fill`.
/// However, a genuine gap of `n` frames must lead to us reverting to `Fill` for `n` frames.
/// To compute this, we use the RTP timestamp of two `seq`-adjacent packets at playout: if the next
/// timestamp is too large, then we revert to `Fill`.
///
/// Small playout bursts also require care.
///
/// If timestamp info is incorrect, then in the worst case we eventually need to rebuffer if the delay
/// drains to zero.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum PlayoutMode {
Fill,
Drain,
}
#[derive(Clone, Debug, Eq, PartialEq)]
pub enum PacketLookup {
Packet(StoredPacket),
MissedPacket,
Filling,
}
#[derive(Debug)]
pub struct PlayoutBuffer {
buffer: VecDeque<Option<StoredPacket>>,
playout_mode: PlayoutMode,
next_seq: RtpSequence,
current_timestamp: Option<RtpTimestamp>,
consecutive_store_fails: usize,
}
impl PlayoutBuffer {
pub fn new(capacity: usize, next_seq: RtpSequence) -> Self {
Self {
buffer: VecDeque::with_capacity(capacity),
playout_mode: PlayoutMode::Fill,
next_seq,
current_timestamp: None,
consecutive_store_fails: 0,
}
}
/// Slot a received RTP packet into the correct location in the playout buffer using
/// its sequence number, subject to maximums.
///
/// An out of bounds packet must create any remaining `None`s
pub fn store_packet(&mut self, packet: StoredPacket, config: &Config) {
let rtp = RtpPacket::new(&packet.packet)
.expect("FATAL: earlier valid packet now invalid (store)");
if self.current_timestamp.is_none() {
self.current_timestamp = Some(reset_timeout(&rtp, config));
}
// compute index by taking wrapping difference between both seq numbers.
// If the difference is *too big*, or in the past [also too big, in a way],
// ignore the packet
let pkt_seq = rtp.get_sequence().0;
let desired_index = (pkt_seq - self.next_seq).0 as i16;
// Similar concept to fetch_packet -- if there's a critical desync, and we're unwilling
// to slot this packet into an empty/stuck buffer then behave as though this packet is the next
// sequence number we're releasing.
let err_threshold = i16::from(config.playout_buffer_length.get() * 5);
let handling_desync = (self.buffer.is_empty()
|| self.consecutive_store_fails >= (err_threshold as usize))
&& desired_index >= err_threshold;
if desired_index < 0 {
trace!("Missed packet arrived late, discarding from playout.");
} else if !handling_desync && desired_index >= 64 {
trace!(
"Packet arrived beyond playout max length({}): wanted slot {desired_index}.\
ts {}, seq {}, next_out_seq {}",
rtp.get_ssrc(),
rtp.get_timestamp().0,
rtp.get_sequence().0,
self.next_seq,
);
self.consecutive_store_fails += 1;
} else {
let index = if handling_desync {
self.buffer.clear();
self.next_seq = pkt_seq;
0
} else {
desired_index as usize
};
while self.buffer.len() <= index {
self.buffer.push_back(None);
}
self.buffer[index] = Some(packet);
self.consecutive_store_fails = 0;
}
if self.buffer.len() >= usize::from(config.playout_buffer_length.get()) {
self.playout_mode = PlayoutMode::Drain;
}
}
pub fn fetch_packet(&mut self, config: &Config) -> PacketLookup {
if self.playout_mode == PlayoutMode::Fill {
return PacketLookup::Filling;
}
let out = match self.buffer.pop_front() {
Some(Some(pkt)) => {
let rtp = RtpPacket::new(&pkt.packet)
.expect("FATAL: earlier valid packet now invalid (fetch)");
// The curr_ts captures the current playout point; we want to
// be able to emit *all* packets with a smaller timestamp.
// However, we need to handle this in a wrap-safe way.
// ts_diff shows where the current time lies if we treat packet_ts
// as 0, s.t. ts_diff >= 0 (equiv) packet_time <= curr_time.
let curr_ts = self.current_timestamp.as_mut().unwrap();
let pkt_ts = rtp.get_timestamp().0;
let ts_diff = (*curr_ts - pkt_ts).0 as i32;
// At least one client in the wild has seen unusual timestamp behaviour: the
// first packet sent out in a run of audio may have an older timestamp.
// This could be badly timestamped, or could conceivably be an orphaned packet
// from a prior run, or e.g.:
// (n x RTP) -> [>100ms delay] -> (RTP) -> [long O(s) delay] -> (m x RTP)
// This leaves us with two adjacent packets in the same playout with wildly varying
// timestamps. We have a slightly tricky situation -- we need to preserve accurate
// timing to correctly drain/refill/recreate very small pauses in audio, but don't
// want to block indefinitely.
//
// We have a compromise here -- if an adjacent (Drain) packet has a ts gap
// larger than it would take to go through multiple Fill/Drain cycles, then
// treat its TS as the next expected value to avoid jamming the buffer and losing
// later audio.
let skip_after = i32::try_from(
usize::from(config.playout_buffer_length.get()) * 5 * MONO_FRAME_SIZE,
)
.unwrap_or((AUDIO_FRAME_RATE * 2 * MONO_FRAME_SIZE) as i32);
if ts_diff >= 0 {
// At or before expected timestamp.
self.next_seq = (rtp.get_sequence() + 1).0;
PacketLookup::Packet(pkt)
} else if ts_diff <= -skip_after {
// >5 playouts ahead.
self.next_seq = (rtp.get_sequence() + 1).0;
*curr_ts = pkt_ts;
PacketLookup::Packet(pkt)
} else {
trace!("Witholding packet: ts_diff is {ts_diff}");
self.buffer.push_front(Some(pkt));
self.playout_mode = PlayoutMode::Fill;
PacketLookup::Filling
}
},
Some(None) => {
self.next_seq += 1;
PacketLookup::MissedPacket
},
None => PacketLookup::Filling,
};
if self.buffer.is_empty() {
self.playout_mode = PlayoutMode::Fill;
self.current_timestamp = None;
}
if let Some(ts) = self.current_timestamp.as_mut() {
*ts += &(MONO_FRAME_SIZE as u32);
}
out
}
pub fn next_seq(&self) -> RtpSequence {
self.next_seq
}
}
#[inline]
fn reset_timeout(packet: &RtpPacket<'_>, config: &Config) -> RtpTimestamp {
let t_shift = MONO_FRAME_SIZE * usize::from(config.playout_buffer_length.get());
(packet.get_timestamp() + (t_shift as u32)).0
}

View File

@@ -0,0 +1,193 @@
use super::*;
use crate::{
driver::{
tasks::error::{Error, Result},
Channels,
DecodeConfig,
DecodeMode,
},
events::context_data::{RtpData, VoiceData},
};
use discortp::{rtp::RtpExtensionPacket, Packet, PacketSize};
use opus2::{Decoder as OpusDecoder, ErrorCode};
use tracing::{error, warn};
#[derive(Debug)]
pub struct SsrcState {
playout_buffer: PlayoutBuffer,
crypto_mode: CryptoMode,
decoder: OpusDecoder,
decode_size: PacketDecodeSize,
pub(crate) prune_time: Instant,
pub(crate) disconnected: bool,
channels: Channels,
}
impl SsrcState {
pub fn new(pkt: &RtpPacket<'_>, crypto_mode: CryptoMode, config: &Config) -> Self {
let playout_capacity = config.playout_buffer_length.get() + config.playout_spike_length;
let (sample_rate, channels) = match config.decode_mode {
DecodeMode::Decode(config) => (config.sample_rate, config.channels),
DecodeMode::Decrypt | DecodeMode::Pass => Default::default(),
};
Self {
playout_buffer: PlayoutBuffer::new(usize::from(playout_capacity), pkt.get_sequence().0),
crypto_mode,
decoder: OpusDecoder::new(sample_rate.into(), channels.into())
.expect("Failed to create new Opus decoder for source."),
decode_size: PacketDecodeSize::TwentyMillis,
prune_time: Instant::now() + config.decode_state_timeout.into(),
disconnected: false,
channels,
}
}
pub fn reconfigure_decoder(&mut self, config: DecodeConfig) {
self.decoder = OpusDecoder::new(config.sample_rate.into(), config.channels.into())
.expect("Failed to create new Opus decoder for source.");
self.channels = config.channels;
}
pub fn store_packet(&mut self, packet: StoredPacket, config: &Config) {
self.playout_buffer.store_packet(packet, config);
}
pub fn refresh_timer(&mut self, state_timeout: Duration) {
if !self.disconnected {
self.prune_time = Instant::now() + state_timeout;
}
}
pub fn get_voice_tick(&mut self, config: &Config) -> Result<Option<VoiceData>> {
// Acquire a packet from the playout buffer:
// Update nexts, lasts...
// different cases: null packet who we want to decode as a miss, and packet who we must ignore temporarily.
let m_pkt = self.playout_buffer.fetch_packet(config);
let pkt = match m_pkt {
PacketLookup::Packet(StoredPacket { packet, decrypted }) => Some((packet, decrypted)),
PacketLookup::MissedPacket => None,
PacketLookup::Filling => return Ok(None),
};
let mut out = VoiceData {
packet: None,
decoded_voice: None,
};
let should_decode = config.decode_mode.should_decode();
if let Some((packet, decrypted)) = pkt {
let rtp = RtpPacket::new(&packet).unwrap();
let extensions = rtp.get_extension() != 0;
let payload = rtp.payload();
let payload_offset = self.crypto_mode.payload_prefix_len();
let payload_end_pad = payload.len() - self.crypto_mode.payload_suffix_len();
// We still need to compute missed packets here in case of long loss chains or similar.
// This occurs due to the fallback in 'store_packet' (i.e., empty buffer and massive seq difference).
// Normal losses should be handled by the below `else` branch.
let new_seq: u16 = rtp.get_sequence().into();
let missed_packets = new_seq.saturating_sub(self.playout_buffer.next_seq().0);
// TODO: maybe hand over audio and extension indices alongside packet?
let (audio, _packet_size) = self.scan_and_decode(
&payload[payload_offset..payload_end_pad],
extensions,
missed_packets,
should_decode && decrypted,
)?;
let rtp_data = RtpData {
packet,
payload_offset,
payload_end_pad,
};
out.packet = Some(rtp_data);
out.decoded_voice = audio;
} else if should_decode {
let mut audio = vec![0; self.decode_size.len()];
let dest_samples = (&mut audio[..])
.try_into()
.expect("Decode logic will cap decode buffer size at i32::MAX.");
let len = self.decoder.decode(&[], dest_samples, false)?;
audio.truncate(2 * len);
out.decoded_voice = Some(audio);
}
Ok(Some(out))
}
fn scan_and_decode(
&mut self,
data: &[u8],
extension: bool,
missed_packets: u16,
decode: bool,
) -> Result<(Option<Vec<i16>>, usize)> {
let start = if extension {
RtpExtensionPacket::new(data)
.map(|pkt| pkt.packet_size())
.ok_or_else(|| {
error!("Extension packet indicated, but insufficient space.");
Error::IllegalVoicePacket
})
} else {
Ok(0)
}?;
let pkt = if decode {
let mut out = vec![0; self.decode_size.len()];
for _ in 0..missed_packets {
let dest_samples = (&mut out[..])
.try_into()
.expect("Decode logic will cap decode buffer size at i32::MAX.");
if let Err(e) = self.decoder.decode(&[], dest_samples, false) {
warn!("Issue while decoding for missed packet: {:?}.", e);
}
}
// In general, we should expect 20 ms frames.
// However, Discord occasionally like to surprise us with something bigger.
// This is *sender-dependent behaviour*.
//
// This should scan up to find the "correct" size that a source is using,
// and then remember that.
loop {
let tried_audio_len = self.decoder.decode(&data[start..], &mut out, false);
match tried_audio_len {
Ok(audio_len) => {
// Decoding to stereo: audio_len refers to sample count irrespective of channel count.
// => multiply by number of channels.
out.truncate(self.channels.channels() * audio_len);
break;
},
Err(e) if e.code() == ErrorCode::BufferTooSmall => {
if self.decode_size.can_bump_up() {
self.decode_size = self.decode_size.bump_up();
out = vec![0; self.decode_size.len()];
} else {
error!("Received packet larger than Opus standard maximum,");
return Err(Error::IllegalVoicePacket);
}
},
Err(e) => {
error!("Failed to decode received packet: {:?}.", e);
return Err(e.into());
},
}
}
Some(out)
} else {
None
};
Ok((pkt, data.len() - start))
}
}

View File

@@ -0,0 +1,559 @@
use super::message::*;
use crate::{
driver::tasks::error::DaveReinitError,
events::CoreContext,
model::{
payload::{Heartbeat, Speaking},
CloseCode as VoiceCloseCode,
Event as GatewayEvent,
FromPrimitive,
SpeakingState,
},
ws::{Error as WsError, WsStream},
ConnectionInfo,
};
use flume::Receiver;
use rand::{distr::Uniform, Rng};
use serenity_voice_model::{
id::UserId,
payload::{
DaveMlsCommitWelcome,
DaveMlsInvalidCommitWelcome,
DaveMlsKeyPackage,
DaveMlsProposalsOperationType,
DaveTransitionReady,
},
};
use std::{
collections::{HashMap, HashSet},
num::NonZeroU16,
sync::{
atomic::{AtomicU16, Ordering},
Arc,
RwLock,
},
time::Duration,
};
use tokio::{
select,
time::{sleep_until, Instant},
};
#[cfg(feature = "tungstenite")]
use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode;
use tracing::{debug, info, instrument, trace, warn};
pub(crate) struct AuxNetwork {
rx: Receiver<WsMessage>,
ws_client: WsStream,
dont_send: bool,
ssrc: u32,
heartbeat_interval: Duration,
speaking: SpeakingState,
last_heartbeat_nonce: Option<u64>,
attempt_idx: usize,
info: ConnectionInfo,
dave_session: Arc<RwLock<Option<davey::DaveSession>>>,
dave_protocol_version: Arc<AtomicU16>,
dave_pending_transitions: HashMap<u16, u16>,
recognized_user_ids: HashSet<UserId>,
#[cfg(feature = "receive")]
ssrc_signalling: Arc<SsrcTracker>,
}
impl AuxNetwork {
pub(crate) fn new(
evt_rx: Receiver<WsMessage>,
ws_client: WsStream,
ssrc: u32,
heartbeat_interval: f64,
attempt_idx: usize,
info: ConnectionInfo,
dave_session: Arc<RwLock<Option<davey::DaveSession>>>,
dave_protocol_version: Arc<AtomicU16>,
#[cfg(feature = "receive")] ssrc_signalling: Arc<SsrcTracker>,
) -> Self {
let mut recognized_user_ids = HashSet::new();
recognized_user_ids.insert(info.user_id.into());
Self {
rx: evt_rx,
ws_client,
dont_send: false,
ssrc,
heartbeat_interval: Duration::from_secs_f64(heartbeat_interval / 1000.0),
speaking: SpeakingState::empty(),
last_heartbeat_nonce: None,
attempt_idx,
info,
dave_session,
dave_protocol_version,
dave_pending_transitions: HashMap::new(),
recognized_user_ids,
#[cfg(feature = "receive")]
ssrc_signalling,
}
}
#[instrument(skip(self))]
async fn run(&mut self, interconnect: &mut Interconnect) {
let mut next_heartbeat = Instant::now() + self.heartbeat_interval;
loop {
let mut ws_error = false;
let mut should_reconnect = false;
let mut ws_reason = None;
let hb = sleep_until(next_heartbeat);
select! {
// Biased polling (polling from top to bottom) is needed to process WebSocket events
// queued in the initial handshake (messages before SessionDescription). One of the
// events queued is ClientsConnect, which is needed to correctly keep track of
// recognized_user_ids and to correctly process DaveMlsProposals.
biased;
() = hb => {
ws_error = match self.send_heartbeat().await {
Err(e) => {
should_reconnect = ws_error_is_not_final(&e);
ws_reason = Some((&e).into());
true
},
_ => false,
};
next_heartbeat = self.next_heartbeat();
}
inner_msg = self.rx.recv_async() => {
match inner_msg {
Ok(WsMessage::Ws(data)) => {
self.ws_client = *data;
next_heartbeat = self.next_heartbeat();
self.dont_send = false;
},
Ok(WsMessage::ReplaceInterconnect(i)) => {
*interconnect = i;
},
Ok(WsMessage::SetKeepalive(keepalive)) => {
self.heartbeat_interval = Duration::from_secs_f64(keepalive / 1000.0);
next_heartbeat = self.next_heartbeat();
},
Ok(WsMessage::Speaking(is_speaking)) => {
if self.speaking.contains(SpeakingState::MICROPHONE) != is_speaking && !self.dont_send {
self.speaking.set(SpeakingState::MICROPHONE, is_speaking);
info!("Changing to {:?}", self.speaking);
let ssu_status = self.ws_client
.send_json(&GatewayEvent::from(Speaking {
delay: Some(0),
speaking: self.speaking,
ssrc: self.ssrc,
user_id: None,
}))
.await;
ws_error |= match ssu_status {
Err(e) => {
should_reconnect = ws_error_is_not_final(&e);
ws_reason = Some((&e).into());
true
},
_ => false,
}
}
},
Ok(WsMessage::Deliver(msg)) => {
ws_error |= match self.process_ws(interconnect, msg).await {
Err(e) => {
should_reconnect = ws_error_is_not_final(&e);
ws_reason = Some((&e).into());
true
}
_ => false,
}
},
Err(flume::RecvError::Disconnected) => {
break;
},
}
}
ws_msg = self.ws_client.recv_event_no_timeout(), if !self.dont_send => {
ws_error = match ws_msg {
Err(e) => {
should_reconnect = ws_error_is_not_final(&e);
ws_reason = Some((&e).into());
true
},
Ok(Some(msg)) => {
match self.process_ws(interconnect, msg).await {
Err(e) => {
should_reconnect = ws_error_is_not_final(&e);
ws_reason = Some((&e).into());
true
},
_ => false
}
},
_ => false,
};
}
}
if ws_error {
self.dont_send = true;
if should_reconnect {
drop(interconnect.core.send(CoreMessage::Reconnect));
} else {
drop(interconnect.core.send(CoreMessage::SignalWsClosure(
self.attempt_idx,
self.info.clone(),
ws_reason,
)));
break;
}
}
}
}
fn next_heartbeat(&self) -> Instant {
Instant::now() + self.heartbeat_interval
}
async fn send_heartbeat(&mut self) -> Result<(), WsError> {
// Discord have suddenly, mysteriously, started rejecting
// ints-as-strings. Keep JS happy here, I suppose...
const JS_MAX_INT: u64 = (1u64 << 53) - 1;
let nonce_range =
Uniform::new(0, JS_MAX_INT).expect("uniform range is finite and nonempty");
let nonce = rand::rng().sample(nonce_range);
self.last_heartbeat_nonce = Some(nonce);
trace!("Sent heartbeat {:?}", self.speaking);
if !self.dont_send {
self.ws_client
.send_json(&GatewayEvent::from(Heartbeat { nonce }))
.await?;
}
Ok(())
}
async fn process_ws(
&mut self,
interconnect: &Interconnect,
value: GatewayEvent,
) -> Result<(), WsError> {
match value {
GatewayEvent::Speaking(ev) => {
// Jarvis patch: the event goes out before the SSRC is named to the UDP task, so the
// packets that task held for it (udp_rx/held.rs) reach event handlers after it.
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::SpeakingStateUpdate(ev),
)));
#[cfg(feature = "receive")]
if let Some(user_id) = &ev.user_id {
self.ssrc_signalling.user_ssrc_map.insert(*user_id, ev.ssrc);
self.ssrc_signalling.ssrc_user_map.insert(ev.ssrc, *user_id);
}
},
GatewayEvent::ClientConnect(ev) => {
debug!("Received discontinued ClientConnect: {:?}", ev);
},
GatewayEvent::ClientDisconnect(ev) => {
#[cfg(feature = "receive")]
{
self.ssrc_signalling.disconnected_users.insert(ev.user_id);
}
self.recognized_user_ids.remove(&ev.user_id);
drop(interconnect.events.send(EventMessage::FireCoreEvent(
CoreContext::ClientDisconnect(ev),
)));
},
GatewayEvent::ClientsConnect(ev) => {
self.recognized_user_ids.extend(&ev.user_ids);
},
GatewayEvent::HeartbeatAck(ev) => {
if let Some(nonce) = self.last_heartbeat_nonce.take() {
if ev.nonce == nonce {
trace!("Heartbeat ACK received.");
} else {
warn!(
"Heartbeat nonce mismatch! Expected {}, saw {}.",
nonce, ev.nonce
);
}
}
},
GatewayEvent::DavePrepareTransition(ev) => {
self.dave_pending_transitions
.insert(ev.transition_id, ev.protocol_version);
if ev.transition_id == 0 {
self.execute_dave_transition(ev.transition_id);
} else if ev.protocol_version == 0 {
if let Some(ref mut dave_session) = *self.dave_session.write().unwrap() {
dave_session.set_passthrough_mode(true, Some(120));
}
self.ws_client
.send_json(&GatewayEvent::from(DaveTransitionReady {
transition_id: ev.transition_id,
protocol_version: ev.protocol_version,
}))
.await?;
}
},
GatewayEvent::DaveExecuteTransition(ev) => {
self.execute_dave_transition(ev.transition_id);
},
GatewayEvent::DavePrepareEpoch(ev) if ev.epoch == 1 => {
self.dave_protocol_version
.store(ev.protocol_version, Ordering::Relaxed);
match self.reinit_dave_session().await {
Err(DaveReinitError::Ws(e)) => return Err(e),
Err(e) => {
warn!(error = ?e, "failed to reinitialize DAVE session");
},
_ => {},
}
},
GatewayEvent::DaveMlsExternalSender(ev) => {
if let Some(ref mut dave_session) = *self.dave_session.write().unwrap() {
if let Err(e) = dave_session.set_external_sender(&ev.external_sender) {
warn!(error = ?e, "error setting MLS external sender");
}
}
},
GatewayEvent::DaveMlsProposals(ev) => {
let operation_type = match ev.operation_type {
DaveMlsProposalsOperationType::Append => davey::ProposalsOperationType::APPEND,
DaveMlsProposalsOperationType::Revoke => davey::ProposalsOperationType::REVOKE,
};
let result = if let Some(ref mut dave_session) = *self.dave_session.write().unwrap()
{
match dave_session.process_proposals(
operation_type,
&ev.proposals,
Some(
&self
.recognized_user_ids
.iter()
.map(|u| u.0)
.collect::<Vec<_>>(),
),
) {
Ok(result) => result,
Err(e) => {
warn!(error = ?e, "error processing MLS proposals");
None
},
}
} else {
None
};
if let Some(commit_welcome) = result {
self.ws_client
.send_binary(&GatewayEvent::from(DaveMlsCommitWelcome {
commit: commit_welcome.commit,
welcome: commit_welcome.welcome,
}))
.await?;
}
},
GatewayEvent::DaveMlsAnnounceCommitTransition(ev) => {
match self.dave_process_commit(&ev.commit_message) {
Some(Ok(())) if ev.transition_id != 0 => {
let protocol_version = self.dave_protocol_version.load(Ordering::Relaxed);
self.dave_pending_transitions
.insert(ev.transition_id, protocol_version);
self.ws_client
.send_json(&GatewayEvent::from(DaveTransitionReady {
transition_id: ev.transition_id,
protocol_version,
}))
.await?;
},
Some(Err(e)) => {
warn!("MLS commit errored: {e:?}");
self.ws_client
.send_json(&GatewayEvent::from(DaveMlsInvalidCommitWelcome {
transition_id: ev.transition_id,
}))
.await?;
match self.reinit_dave_session().await {
Err(DaveReinitError::Ws(e)) => return Err(e),
Err(e) => {
warn!(error = ?e, "failed to reinitialize DAVE session");
},
_ => {},
}
},
Some(Ok(())) | None => {},
}
},
GatewayEvent::DaveMlsWelcome(ev) => match self.dave_process_welcome(&ev.welcome) {
Some(Ok(())) if ev.transition_id != 0 => {
let protocol_version = self.dave_protocol_version.load(Ordering::Relaxed);
self.dave_pending_transitions
.insert(ev.transition_id, protocol_version);
self.ws_client
.send_json(&GatewayEvent::from(DaveTransitionReady {
transition_id: ev.transition_id,
protocol_version,
}))
.await?;
},
Some(Err(e)) => {
warn!("MLS welcome errored: {e:?}");
self.ws_client
.send_json(&GatewayEvent::from(DaveMlsInvalidCommitWelcome {
transition_id: ev.transition_id,
}))
.await?;
match self.reinit_dave_session().await {
Err(DaveReinitError::Ws(e)) => return Err(e),
Err(e) => {
warn!(error = ?e, "failed to reinitialize DAVE session");
},
_ => {},
}
},
Some(Ok(())) | None => {},
},
other => {
trace!("Received other websocket data: {:?}", other);
},
}
Ok(())
}
fn dave_process_commit(
&mut self,
commit_message: &[u8],
) -> Option<Result<(), davey::errors::ProcessCommitError>> {
let mut dave_session_lock = self.dave_session.write().unwrap();
let dave_session = (*dave_session_lock).as_mut()?;
Some(dave_session.process_commit(commit_message))
}
fn dave_process_welcome(
&mut self,
welcome: &[u8],
) -> Option<Result<(), davey::errors::ProcessWelcomeError>> {
let mut dave_session_lock = self.dave_session.write().unwrap();
let dave_session = (*dave_session_lock).as_mut()?;
Some(dave_session.process_welcome(welcome))
}
async fn reinit_dave_session(&mut self) -> Result<(), DaveReinitError> {
let protocol_version = self.dave_protocol_version.load(Ordering::Relaxed);
if let Some(dave_protocol_version) = NonZeroU16::new(protocol_version) {
let user_id = self.info.user_id.0.into();
let channel_id = self.info.channel_id.0.into();
let key_package =
if let Some(ref mut dave_session) = *self.dave_session.write().unwrap() {
dave_session.reinit(dave_protocol_version, user_id, channel_id, None)?;
dave_session.create_key_package()?
} else {
let mut dave_session =
davey::DaveSession::new(dave_protocol_version, user_id, channel_id, None)?;
let key_package = dave_session.create_key_package()?;
*self.dave_session.write().unwrap() = Some(dave_session);
key_package
};
self.ws_client
.send_binary(&GatewayEvent::DaveMlsKeyPackage(DaveMlsKeyPackage {
key_package,
}))
.await?;
} else if let Some(ref mut dave_session) = *self.dave_session.write().unwrap() {
dave_session.reset()?;
dave_session.set_passthrough_mode(true, Some(10));
}
Ok(())
}
fn execute_dave_transition(&mut self, transition_id: u16) {
let Some(new_version) = self.dave_pending_transitions.get(&transition_id).copied() else {
warn!("Received DaveExecuteTransition for unknown transition ID {transition_id}");
return;
};
let old_version = self.dave_protocol_version.load(Ordering::Relaxed);
self.dave_protocol_version
.store(new_version, Ordering::Relaxed);
// Upgraded from transport-only encryption
if transition_id > 0 && old_version == 0 && new_version != 0 {
if let Some(ref mut dave_session) = *self.dave_session.write().unwrap() {
dave_session.set_passthrough_mode(true, Some(10));
}
}
self.dave_pending_transitions.remove(&transition_id);
}
}
#[instrument(skip(interconnect, aux))]
pub(crate) async fn runner(mut interconnect: Interconnect, mut aux: AuxNetwork) {
trace!("WS thread started.");
aux.run(&mut interconnect).await;
trace!("WS thread finished.");
}
fn ws_error_is_not_final(err: &WsError) -> bool {
match err {
#[cfg(feature = "tungstenite")]
WsError::WsClosed(Some(frame)) => match frame.code {
CloseCode::Library(l) =>
if let Some(code) = VoiceCloseCode::from_u16(l) {
code.should_resume()
} else {
true
},
_ => true,
},
#[cfg(feature = "tws")]
WsError::WsClosed(Some(code)) => match (*code).into() {
code @ 4000..=4999_u16 =>
if let Some(code) = VoiceCloseCode::from_u16(code) {
code.should_resume()
} else {
true
},
_ => true,
},
e => {
debug!("Error sending/receiving ws {:?}.", e);
true
},
}
}

View File

@@ -0,0 +1,267 @@
#![allow(missing_docs)]
use flume::{Receiver, Sender};
use crate::{
tracks::{PlayMode, TrackHandle, TrackState},
Event,
EventContext,
EventHandler,
TrackEvent,
};
use std::time::Duration;
#[allow(dead_code)]
#[derive(Clone, Debug)]
pub enum TickStyle {
Timed,
UntimedWithExecLimit(Receiver<u64>),
}
#[derive(Clone, Debug, PartialEq)]
pub enum OutputMessage {
Passthrough(Vec<u8>),
Mixed(Vec<f32>),
Silent,
}
#[allow(dead_code)]
impl OutputMessage {
#[must_use]
pub fn is_passthrough(&self) -> bool {
matches!(self, Self::Passthrough(_))
}
#[must_use]
pub fn is_mixed(&self) -> bool {
matches!(self, Self::Mixed(_))
}
#[must_use]
pub fn is_mixed_with_nonzero_signal(&self) -> bool {
if let Self::Mixed(data) = self {
data.iter().any(|v| *v != 0.0f32)
} else {
false
}
}
#[must_use]
pub fn is_explicit_silence(&self) -> bool {
*self == Self::Silent
}
}
#[allow(dead_code)]
#[derive(Clone, Debug)]
pub enum OutputMode {
Raw(Sender<TickMessage<OutputMessage>>),
Rtp(Sender<TickMessage<Vec<u8>>>),
}
#[allow(dead_code)]
#[derive(Clone, Debug)]
pub enum TickMessage<T> {
El(T),
NoEl,
}
impl<T> From<T> for TickMessage<T> {
fn from(val: T) -> Self {
TickMessage::El(val)
}
}
impl From<TickMessage<OutputMessage>> for OutputPacket {
fn from(val: TickMessage<OutputMessage>) -> Self {
match val {
TickMessage::El(e) => OutputPacket::Raw(e),
TickMessage::NoEl => OutputPacket::Empty,
}
}
}
impl From<TickMessage<Vec<u8>>> for OutputPacket {
fn from(val: TickMessage<Vec<u8>>) -> Self {
match val {
TickMessage::El(e) => OutputPacket::Rtp(e),
TickMessage::NoEl => OutputPacket::Empty,
}
}
}
#[derive(Clone, Debug, PartialEq)]
pub enum OutputPacket {
Raw(OutputMessage),
Rtp(Vec<u8>),
Empty,
}
impl OutputPacket {
#[must_use]
pub fn raw(&self) -> Option<&OutputMessage> {
if let Self::Raw(o) = self {
Some(o)
} else {
None
}
}
}
#[derive(Clone, Debug)]
pub enum OutputReceiver {
Raw(Receiver<TickMessage<OutputMessage>>),
Rtp(Receiver<TickMessage<Vec<u8>>>),
}
#[derive(Clone)]
pub struct DriverTestHandle {
pub rx: OutputReceiver,
pub tx: Sender<u64>,
}
impl DriverTestHandle {
#[must_use]
pub fn recv(&self) -> OutputPacket {
match &self.rx {
OutputReceiver::Raw(rx) => rx.recv().unwrap().into(),
OutputReceiver::Rtp(rx) => rx.recv().unwrap().into(),
}
}
pub async fn recv_async(&self) -> OutputPacket {
match &self.rx {
OutputReceiver::Raw(rx) => rx.recv_async().await.unwrap().into(),
OutputReceiver::Rtp(rx) => rx.recv_async().await.unwrap().into(),
}
}
#[must_use]
pub fn len(&self) -> usize {
match &self.rx {
OutputReceiver::Raw(rx) => rx.len(),
OutputReceiver::Rtp(rx) => rx.len(),
}
}
#[must_use]
pub fn is_empty(&self) -> bool {
self.len() == 0
}
pub fn wait(&self, n_ticks: u64) {
for _i in 0..n_ticks {
drop(self.recv());
}
}
pub async fn wait_async(&self, n_ticks: u64) {
for _i in 0..n_ticks {
drop(self.recv_async().await);
}
}
pub fn spawn_ticker(&self) {
let remote = self.clone();
tokio::spawn(async move {
loop {
remote.skip(1).await;
tokio::time::sleep(Duration::from_millis(1)).await;
}
});
}
pub fn wait_noisy(&self, n_ticks: u64) {
for _i in 0..n_ticks {
match self.recv() {
OutputPacket::Empty => eprintln!("pkt: Nothing"),
OutputPacket::Rtp(p) => eprintln!("pkt: RTP[{}B]", p.len()),
OutputPacket::Raw(OutputMessage::Silent) => eprintln!("pkt: Raw-Silent"),
OutputPacket::Raw(OutputMessage::Passthrough(p)) => {
eprintln!("pkt: Raw-Passthrough[{}B]", p.len());
},
OutputPacket::Raw(OutputMessage::Mixed(p)) => {
eprintln!("pkt: Raw-Mixed[{}B]", p.len());
},
}
}
}
pub async fn skip(&self, n_ticks: u64) {
self.tick(n_ticks);
self.wait_async(n_ticks).await;
}
pub fn tick(&self, n_ticks: u64) {
assert!(
n_ticks != 0,
"Number of ticks to advance driver/mixer must be >= 1."
);
self.tx.send(n_ticks).unwrap();
}
pub async fn ready_track(
&self,
handle: &TrackHandle,
tick_wait: Option<Duration>,
) -> TrackState {
struct SongPlayable {
tx: Sender<TrackState>,
}
#[async_trait::async_trait]
impl EventHandler for SongPlayable {
async fn act(&self, ctx: &crate::EventContext<'_>) -> Option<Event> {
if let EventContext::Track(&[(state, _)]) = ctx {
drop(self.tx.send(state.clone()));
}
Some(Event::Cancel)
}
}
struct SongErred {
tx: Sender<PlayMode>,
}
#[async_trait::async_trait]
impl EventHandler for SongErred {
async fn act(&self, ctx: &crate::EventContext<'_>) -> Option<Event> {
if let EventContext::Track(&[(state, _)]) = ctx {
drop(self.tx.send(state.playing.clone()));
}
Some(Event::Cancel)
}
}
let (tx, rx) = flume::bounded(1);
let (err_tx, err_rx) = flume::bounded(1);
handle
.add_event(Event::Track(TrackEvent::Playable), SongPlayable { tx })
.expect("Adding track evt should not fail before any ticks.");
handle
.add_event(Event::Track(TrackEvent::Error), SongErred { tx: err_tx })
.expect("Adding track evt should not fail before any ticks.");
loop {
self.tick(1);
tokio::time::sleep(tick_wait.unwrap_or_else(|| Duration::from_millis(20))).await;
self.wait_async(1).await;
match err_rx.try_recv() {
Ok(e) => panic!("Error reported on track: {e:?}"),
Err(flume::TryRecvError::Empty | flume::TryRecvError::Disconnected) => {},
}
match rx.try_recv() {
Ok(val) => return val,
Err(flume::TryRecvError::Disconnected) => panic!(),
Err(flume::TryRecvError::Empty) => {},
}
}
}
}

View File

@@ -0,0 +1,255 @@
#![allow(missing_docs)]
use super::{
scheduler::*,
tasks::{message::*, mixer::Mixer},
*,
};
use crate::{
constants::*,
input::{
cached::Compressed,
codecs::{get_codec_registry, get_probe},
RawAdapter,
},
test_utils,
tracks::LoopState,
};
use flume::Receiver;
use std::{
io::Cursor,
net::UdpSocket,
num::NonZeroU16,
sync::{atomic::AtomicU16, Arc, RwLock},
};
use tokio::runtime::Handle;
// create a dummied task + interconnect.
// measure perf at varying numbers of sources (binary 1--64) without passthrough support.
#[cfg(feature = "receive")]
pub type Listeners = (
Receiver<CoreMessage>,
Receiver<EventMessage>,
Receiver<UdpRxMessage>,
);
#[cfg(not(feature = "receive"))]
pub type Listeners = (Receiver<CoreMessage>, Receiver<EventMessage>);
pub type DummyMixer = (Mixer, Listeners);
impl Mixer {
#[must_use]
pub fn mock(handle: Handle, softclip: bool) -> DummyMixer {
let (mix_tx, mix_rx) = flume::unbounded();
let (core_tx, core_rx) = flume::unbounded();
let (event_tx, event_rx) = flume::unbounded();
#[cfg(feature = "receive")]
let (udp_receiver_tx, udp_receiver_rx) = flume::unbounded();
let ic = Interconnect {
core: core_tx,
events: event_tx,
mixer: mix_tx,
};
// Scheduler must be created from a Tokio context...
let (tx, rx) = flume::unbounded();
handle.spawn_blocking(move || tx.send(crate::Config::default().use_softclip(softclip)));
let config = rx.recv().unwrap();
let mut out = Mixer::new(mix_rx, handle, ic, config);
let udp_tx = UdpSocket::bind("0.0.0.0:0").expect("Failed to create send port.");
udp_tx
.connect("127.0.0.1:5316")
.expect("Failed to connect to local dest port.");
let mode = CryptoMode::Aes256Gcm;
let cipher = mode.cipher_from_key(&[0u8; 32]).unwrap();
let crypto_state = mode.into();
let dave_protocol_version = Arc::new(AtomicU16::new(davey::DAVE_PROTOCOL_VERSION));
let dave_session = Arc::new(RwLock::new(Some(
davey::DaveSession::new(
NonZeroU16::new(1).expect("failed to initialize NonZeroU16 from static value"),
1,
1,
None,
)
.expect("failed to initialize davey::DaveSession"),
)));
#[cfg(feature = "receive")]
let fake_conn = MixerConnection {
cipher,
crypto_state,
dave_session,
dave_protocol_version,
udp_rx: udp_receiver_tx,
udp_tx,
};
#[cfg(not(feature = "receive"))]
let fake_conn = MixerConnection {
cipher,
crypto_state,
dave_session,
dave_protocol_version,
udp_tx,
};
out.conn_active = Some(fake_conn);
#[cfg(feature = "receive")]
return (out, (core_rx, event_rx, udp_receiver_rx));
#[cfg(not(feature = "receive"))]
return (out, (core_rx, event_rx));
}
#[must_use]
pub fn test_with_float(num_tracks: usize, handle: Handle, softclip: bool) -> DummyMixer {
let mut out = Self::mock(handle, softclip);
let floats = test_utils::make_sine(10 * STEREO_FRAME_SIZE, true);
for _ in 0..num_tracks {
let input: Input = RawAdapter::new(Cursor::new(floats.clone()), 48_000, 2).into();
let promoted = match input {
Input::Live(l, _) => l.promote(get_codec_registry(), get_probe()),
Input::Lazy(_) => panic!("Failed to create a guaranteed source."),
};
let (_, ctx) = Track::from(Input::Live(promoted.unwrap(), None)).into_context();
_ = out.0.add_track(ctx);
}
out
}
#[must_use]
pub fn test_with_float_unending(handle: Handle, softclip: bool) -> (DummyMixer, TrackHandle) {
let mut out = Self::mock(handle, softclip);
let floats = test_utils::make_sine(10 * STEREO_FRAME_SIZE, true);
let input: Input = RawAdapter::new(Cursor::new(floats.clone()), 48_000, 2).into();
let promoted = match input {
Input::Live(l, _) => l.promote(get_codec_registry(), get_probe()),
Input::Lazy(_) => panic!("Failed to create a guaranteed source."),
};
let mut track = Track::from(Input::Live(promoted.unwrap(), None));
track.loops = LoopState::Infinite;
let (handle, ctx) = track.into_context();
_ = out.0.add_track(ctx);
(out, handle)
}
#[must_use]
pub fn test_with_float_drop(num_tracks: usize, handle: Handle) -> DummyMixer {
let mut out = Self::mock(handle, true);
for i in 0..num_tracks {
let floats = test_utils::make_sine((i / 5) * STEREO_FRAME_SIZE, true);
let input: Input = RawAdapter::new(Cursor::new(floats.clone()), 48_000, 2).into();
let promoted = match input {
Input::Live(l, _) => l.promote(get_codec_registry(), get_probe()),
Input::Lazy(_) => panic!("Failed to create a guaranteed source."),
};
let (_, ctx) = Track::from(Input::Live(promoted.unwrap(), None)).into_context();
_ = out.0.add_track(ctx);
}
out
}
#[must_use]
pub fn test_with_opus(handle: &Handle) -> DummyMixer {
// should add a single opus-based track.
// make this fully loaded to prevent any perf cost there.
let mut out = Self::mock(handle.clone(), false);
let floats = test_utils::make_sine(6 * STEREO_FRAME_SIZE, true);
let input: Input = RawAdapter::new(Cursor::new(floats), 48_000, 2).into();
let mut src = handle.block_on(async move {
Compressed::new(input, Bitrate::Bits(128_000))
.await
.expect("These parameters are well-defined.")
});
src.raw.load_all();
let promoted = match src.into() {
Input::Live(l, _) => l.promote(get_codec_registry(), get_probe()),
Input::Lazy(_) => panic!("Failed to create a guaranteed source."),
};
let (_, ctx) = Track::from(Input::Live(promoted.unwrap(), None)).into_context();
_ = out.0.add_track(ctx);
out
}
}
pub struct MockScheduler {
pub core: Live,
pub stats: Arc<StatBlock>,
pub local: Arc<LiveStatBlock>,
pub rx: Receiver<SchedulerMessage>,
pub tx: Sender<(TaskId, ParkedMixer)>,
pub id: TaskId,
}
impl MockScheduler {
#[must_use]
pub fn new(mode: Option<Mode>) -> Self {
let stats = Arc::new(StatBlock::default());
let local = Arc::new(LiveStatBlock::default());
let (task_tx, task_rx) = flume::unbounded();
let (sched_tx, sched_rx) = flume::unbounded();
let cfg = crate::driver::SchedulerConfig {
strategy: mode.unwrap_or_default(),
move_expensive_tasks: true,
};
let core = Live::new(
WorkerId::new(),
cfg,
stats.clone(),
local.clone(),
task_rx,
sched_tx,
);
Self {
core,
stats,
local,
rx: sched_rx,
tx: task_tx,
id: TaskId::new(),
}
}
pub fn add_mixer_direct(&mut self, m: Mixer) {
let id = self.id.incr();
self.core.add_task_direct(m, id);
}
#[must_use]
pub fn from_mixers(mode: Option<Mode>, mixers: Vec<DummyMixer>) -> (Self, Vec<Listeners>) {
let mut out = Self::new(mode);
let mut listeners = vec![];
for (mixer, listener) in mixers {
out.add_mixer_direct(mixer);
listeners.push(listener);
}
(out, listeners)
}
}

148
app/vendor/songbird/src/error.rs vendored Normal file
View File

@@ -0,0 +1,148 @@
//! Driver and gateway error handling.
#[cfg(feature = "serenity")]
use futures::channel::mpsc::TrySendError;
pub use serde_json::Error as JsonError;
#[cfg(feature = "serenity")]
use serenity::gateway::ShardRunnerMessage;
#[cfg(feature = "gateway")]
use std::{error::Error, fmt};
#[cfg(feature = "twilight")]
use twilight_gateway::error::ChannelError;
#[cfg(feature = "gateway")]
#[derive(Debug)]
#[non_exhaustive]
/// Error returned when a manager or call handler is
/// unable to send messages over Discord's gateway.
pub enum JoinError {
/// Request to join was dropped, cancelled, or replaced.
Dropped,
/// No available gateway connection was provided to send
/// voice state update messages.
NoSender,
/// Tried to leave a [`Call`] which was not found.
///
/// [`Call`]: crate::Call
NoCall,
/// Connection details were not received from Discord in the
/// time given in [the `Call`'s configuration].
///
/// This can occur if a message is lost by the Discord client
/// between restarts, or if Discord's gateway believes that
/// this bot is still in the channel it attempts to join.
///
/// *Users should `leave` the server on the gateway before
/// re-attempting connection.*
///
/// [the `Call`'s configuration]: crate::Config
TimedOut,
#[cfg(feature = "driver")]
/// The driver failed to establish a voice connection.
///
/// *Users should `leave` the server on the gateway before
/// re-attempting connection.*
Driver(ConnectionError),
#[cfg(feature = "serenity")]
/// Serenity-specific WebSocket send error.
Serenity(Box<TrySendError<ShardRunnerMessage>>),
#[cfg(feature = "twilight")]
/// Twilight-specific WebSocket send error when a message fails to send over websocket.
Twilight(ChannelError),
}
#[cfg(feature = "gateway")]
impl JoinError {
/// Indicates whether this failure may have left (or been
/// caused by) Discord's gateway state being in an
/// inconsistent state.
///
/// Failure to `leave` before rejoining may cause further
/// timeouts.
pub fn should_leave_server(&self) -> bool {
matches!(self, JoinError::TimedOut)
}
#[cfg(feature = "driver")]
/// Indicates whether this failure can be reattempted via
/// [`Driver::connect`] with retreived connection info.
///
/// Failure to `leave` before rejoining may cause further
/// timeouts.
///
/// [`Driver::connect`]: crate::driver::Driver
pub fn should_reconnect_driver(&self) -> bool {
matches!(self, JoinError::Driver(_))
}
}
#[cfg(feature = "gateway")]
impl fmt::Display for JoinError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "failed to join voice channel: ")?;
match self {
JoinError::Dropped => write!(f, "request was cancelled/dropped"),
JoinError::NoSender => write!(f, "no gateway destination"),
JoinError::NoCall => write!(f, "tried to leave a non-existent call"),
JoinError::TimedOut => write!(f, "gateway response from Discord timed out"),
#[cfg(feature = "driver")]
JoinError::Driver(_) => write!(f, "establishing connection failed"),
#[cfg(feature = "serenity")]
JoinError::Serenity(e) => e.fmt(f),
#[cfg(feature = "twilight")]
JoinError::Twilight(e) => e.fmt(f),
}
}
}
#[cfg(feature = "gateway")]
impl Error for JoinError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
match self {
JoinError::Dropped => None,
JoinError::NoSender => None,
JoinError::NoCall => None,
JoinError::TimedOut => None,
#[cfg(feature = "driver")]
JoinError::Driver(e) => Some(e),
#[cfg(feature = "serenity")]
JoinError::Serenity(e) => e.source(),
#[cfg(feature = "twilight")]
JoinError::Twilight(e) => e.source(),
}
}
}
#[cfg(all(feature = "serenity", feature = "gateway"))]
impl From<Box<TrySendError<ShardRunnerMessage>>> for JoinError {
fn from(e: Box<TrySendError<ShardRunnerMessage>>) -> Self {
JoinError::Serenity(e)
}
}
#[cfg(all(feature = "twilight", feature = "gateway"))]
impl From<ChannelError> for JoinError {
fn from(e: ChannelError) -> Self {
JoinError::Twilight(e)
}
}
#[cfg(all(feature = "driver", feature = "gateway"))]
impl From<ConnectionError> for JoinError {
fn from(e: ConnectionError) -> Self {
JoinError::Driver(e)
}
}
#[cfg(feature = "gateway")]
/// Convenience type for Discord gateway error handling.
pub type JoinResult<T> = Result<T, JoinError>;
#[cfg(feature = "driver")]
pub use crate::{
driver::{
connection::error::{Error as ConnectionError, Result as ConnectionResult},
SchedulerError,
},
tracks::{ControlError, PlayError, TrackResult},
};

View File

@@ -0,0 +1,31 @@
use crate::id::*;
/// Voice connection details gathered at setup/reinstantiation.
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
#[non_exhaustive]
pub struct ConnectData<'a> {
/// ID of the voice channel being joined.
///
/// This can be used to reconnect/renew a voice session via the gateway.
pub channel_id: ChannelId,
/// ID of the target voice channel's parent guild.
pub guild_id: GuildId,
/// Unique string describing this session for validation/authentication purposes.
pub session_id: &'a str,
/// The domain name of Discord's voice/TURN server.
///
/// With the introduction of Discord's automatic voice server selection,
/// this is no longer guaranteed to match a server's settings. This field
/// may be useful if you need/wish to move your voice connection to a node/shard
/// closer to Discord.
pub server: &'a str,
/// The [RTP SSRC] *("Synchronisation source")* assigned by the voice server
/// for the duration of this call.
///
/// All packets sent will use this SSRC, which is not related to the sender's User
/// ID. These are usually allocated sequentially by Discord, following on from
/// a random starting SSRC.
///
/// [RTP SSRC]: https://tools.ietf.org/html/rfc3550#section-3
pub ssrc: u32,
}

View File

@@ -0,0 +1,126 @@
use crate::{
error::ConnectionError,
id::*,
model::{CloseCode as VoiceCloseCode, FromPrimitive},
ws::Error as WsError,
};
#[cfg(feature = "tungstenite")]
use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode;
/// Voice connection details gathered at termination or failure.
///
/// In the event of a failure, this event data is gathered after
/// a reconnection strategy has exhausted all of its attempts.
#[derive(Debug)]
#[non_exhaustive]
pub struct DisconnectData<'a> {
/// The location that a voice connection was terminated.
pub kind: DisconnectKind,
/// The cause of any connection failure.
///
/// If `None`, then this disconnect was requested by the user in some way
/// (i.e., leaving or changing voice channels).
pub reason: Option<DisconnectReason>,
/// ID of the voice channel being joined.
///
/// This can be used to reconnect/renew a voice session via the gateway.
pub channel_id: ChannelId,
/// ID of the target voice channel's parent guild.
pub guild_id: GuildId,
/// Unique string describing this session for validation/authentication purposes.
pub session_id: &'a str,
}
/// The location that a voice connection was terminated.
#[derive(Copy, Clone, Debug, Eq, Hash, PartialEq)]
#[non_exhaustive]
pub enum DisconnectKind {
/// The voice driver failed to connect to the server.
///
/// This requires explicit handling at the gateway level
/// to either reconnect or fully disconnect.
Connect,
/// The voice driver failed to reconnect to the server.
///
/// This requires explicit handling at the gateway level
/// to either reconnect or fully disconnect.
Reconnect,
/// The voice connection was terminated mid-session by either
/// the user or Discord.
///
/// If `reason == None`, then this disconnection is either
/// a full disconnect or a user-requested channel change.
/// Otherwise, this is likely a session expiry (requiring user
/// handling to fully disconnect/reconnect).
Runtime,
}
/// The reason that a voice connection failed.
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
#[non_exhaustive]
pub enum DisconnectReason {
/// This (re)connection attempt was dropped due to another request.
AttemptDiscarded,
/// Songbird had an internal error.
///
/// This should never happen; if this is ever seen, raise an issue with logs.
Internal,
/// A host-specific I/O error caused the fault; this is likely transient, and
/// should be retried some time later.
Io,
/// Songbird and Discord disagreed on the protocol used to establish a
/// voice connection.
///
/// This should never happen; if this is ever seen, raise an issue with logs.
ProtocolViolation,
/// A voice connection was not established in the specified time.
TimedOut,
/// The call was manually disconnected by a user command, e.g. [`Driver::leave`].
///
/// [`Driver::leave`]: crate::driver::Driver::leave
Requested,
/// The Websocket connection was closed by Discord.
///
/// This typically indicates that the voice session has expired,
/// and a new one needs to be requested via the gateway.
WsClosed(Option<VoiceCloseCode>),
}
impl From<&ConnectionError> for DisconnectReason {
fn from(e: &ConnectionError) -> Self {
match e {
ConnectionError::AttemptDiscarded => Self::AttemptDiscarded,
ConnectionError::CryptoInvalidLength
| ConnectionError::CryptoModeInvalid
| ConnectionError::CryptoModeUnavailable
| ConnectionError::DaveCreateKeyPackageError(_)
| ConnectionError::DaveInitializationError(_)
| ConnectionError::EndpointUrl
| ConnectionError::IllegalDiscoveryResponse
| ConnectionError::IllegalIp
| ConnectionError::Json(_) => Self::ProtocolViolation,
ConnectionError::Io(_) => Self::Io,
ConnectionError::Crypto(_) | ConnectionError::InterconnectFailure(_) => Self::Internal,
ConnectionError::Ws(ws) => ws.into(),
ConnectionError::TimedOut => Self::TimedOut,
}
}
}
impl From<&WsError> for DisconnectReason {
fn from(e: &WsError) -> Self {
Self::WsClosed(match e {
#[cfg(feature = "tungstenite")]
WsError::WsClosed(Some(frame)) => match frame.code {
CloseCode::Library(l) => VoiceCloseCode::from_u16(l),
_ => None,
},
#[cfg(feature = "tws")]
WsError::WsClosed(Some(code)) => match (*code).into() {
code @ 4000..=4999_u16 => VoiceCloseCode::from_u16(code),
_ => None,
},
_ => None,
})
}
}

View File

@@ -0,0 +1,18 @@
//! Types containing the main body of an [`EventContext`].
//!
//! [`EventContext`]: super::EventContext
mod connect;
mod disconnect;
#[cfg(feature = "receive")]
mod rtcp;
#[cfg(feature = "receive")]
mod rtp;
#[cfg(feature = "receive")]
mod voice;
#[cfg(feature = "receive")]
use bytes::Bytes;
pub use self::{connect::*, disconnect::*};
#[cfg(feature = "receive")]
pub use self::{rtcp::*, rtp::*, voice::*};

View File

@@ -0,0 +1,29 @@
use discortp::rtcp::RtcpPacket;
use super::*;
#[derive(Clone, Debug, Eq, PartialEq)]
#[non_exhaustive]
/// Telemetry/statistics packet, received from another stream
///
/// `payload_offset` contains the true payload location within the raw packet's `payload()`,
/// to allow manual decoding of `Rtcp` packet bodies.
pub struct RtcpData {
/// Raw RTCP packet data.
pub packet: Bytes,
/// Byte index into the packet body (after headers) for where the payload begins.
pub payload_offset: usize,
/// Number of bytes at the end of the packet to discard.
pub payload_end_pad: usize,
}
impl RtcpData {
/// Create a zero-copy view of the inner RTCP packet.
///
/// This allows easy access to packet header fields, taking them from the underlying
/// `Bytes` as needed while handling endianness etc.
pub fn rtcp(&'_ self) -> RtcpPacket<'_> {
RtcpPacket::new(&self.packet)
.expect("FATAL: leaked illegally small RTP packet from UDP Rx task.")
}
}

View File

@@ -0,0 +1,31 @@
use discortp::rtp::RtpPacket;
use super::*;
#[derive(Clone, Debug, Eq, PartialEq)]
#[non_exhaustive]
/// Opus audio packet, received from another stream
///
/// `payload_offset` contains the true payload location within the raw packet's `payload()`,
/// if extensions or raw packet data are required.
pub struct RtpData {
/// Raw RTP packet data.
///
/// Includes the SSRC (i.e., sender) of this packet.
pub packet: Bytes,
/// Byte index into the packet body (after headers) for where the payload begins.
pub payload_offset: usize,
/// Number of bytes at the end of the packet to discard.
pub payload_end_pad: usize,
}
impl RtpData {
/// Create a zero-copy view of the inner RTP packet.
///
/// This allows easy access to packet header fields, taking them from the underlying
/// `Bytes` as needed while handling endianness etc.
pub fn rtp(&'_ self) -> RtpPacket<'_> {
RtpPacket::new(&self.packet)
.expect("FATAL: leaked illegally small RTP packet from UDP Rx task.")
}
}

View File

@@ -0,0 +1,43 @@
use std::collections::{HashMap, HashSet};
use super::*;
#[derive(Clone, Debug, Eq, PartialEq)]
#[non_exhaustive]
/// Audio data from all users in a voice channel, fired every 20ms.
///
/// Songbird implements a jitter buffer to sycnhronise user packets, smooth out network latency, and
/// handle packet reordering by the network. Packet playout via this event is delayed by approximately
/// [`Config::playout_buffer_length`]` * 20ms` from its original arrival.
///
/// [`Config::playout_buffer_length`]: crate::Config::playout_buffer_length
pub struct VoiceTick {
/// Decoded voice data and source packets sent by each user.
pub speaking: HashMap<u32, VoiceData>,
/// Set of all SSRCs currently known in the call who aren't included in [`Self::speaking`].
pub silent: HashSet<u32>,
}
#[derive(Clone, Debug, Eq, PartialEq)]
#[non_exhaustive]
/// Voice packet and audio data for a single user, from a single tick.
pub struct VoiceData {
/// RTP packet clocked out for this tick.
///
/// If `None`, then the packet was lost, and [`Self::decoded_voice`] may include
/// around one codec delay's worth of audio.
pub packet: Option<RtpData>,
/// PCM audio obtained from a user.
///
/// Valid audio data (`Some(audio)` where `audio.len >= 0`) typically contains 20ms of 16-bit PCM audio
/// using native endianness. This defaults to stereo audio at 48kHz, and can be configured via
/// [`DecodeConfig::sample_rate`] and [`DecodeConfig::sample_rate`] -- channels are interleaved
/// (i.e., `L, R, L, R, ...`) if stereo.
///
/// This value will be `None` if Songbird is not configured to decode audio.
///
/// [`DecodeConfig::decode_channels`]: crate::driver::DecodeConfig::channels
/// [`DecodeConfig::sample_rate`]: crate::driver::DecodeConfig::sample_rate
pub decoded_voice: Option<Vec<i16>>,
}

View File

@@ -0,0 +1,82 @@
use super::context_data::*;
use crate::ConnectionInfo;
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub struct InternalConnect {
pub info: ConnectionInfo,
pub ssrc: u32,
}
#[derive(Debug)]
pub struct InternalDisconnect {
pub kind: DisconnectKind,
pub reason: Option<DisconnectReason>,
pub info: ConnectionInfo,
}
impl<'a> From<&'a InternalConnect> for ConnectData<'a> {
fn from(val: &'a InternalConnect) -> Self {
Self {
channel_id: val.info.channel_id,
guild_id: val.info.guild_id,
session_id: &val.info.session_id,
server: &val.info.endpoint,
ssrc: val.ssrc,
}
}
}
impl<'a> From<&'a InternalDisconnect> for DisconnectData<'a> {
fn from(val: &'a InternalDisconnect) -> Self {
Self {
kind: val.kind,
reason: val.reason,
channel_id: val.info.channel_id,
guild_id: val.info.guild_id,
session_id: &val.info.session_id,
}
}
}
#[cfg(feature = "receive")]
mod receive {
use super::*;
use bytes::Bytes;
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct InternalRtpPacket {
pub packet: Bytes,
pub payload_offset: usize,
pub payload_end_pad: usize,
}
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct InternalRtcpPacket {
pub packet: Bytes,
pub payload_offset: usize,
pub payload_end_pad: usize,
}
impl<'a> From<&'a InternalRtpPacket> for RtpData {
fn from(val: &'a InternalRtpPacket) -> Self {
Self {
packet: val.packet.clone(),
payload_offset: val.payload_offset,
payload_end_pad: val.payload_end_pad,
}
}
}
impl<'a> From<&'a InternalRtcpPacket> for RtcpData {
fn from(val: &'a InternalRtcpPacket) -> Self {
Self {
packet: val.packet.clone(),
payload_offset: val.payload_offset,
payload_end_pad: val.payload_end_pad,
}
}
}
}
#[cfg(feature = "receive")]
pub use receive::*;

View File

@@ -0,0 +1,114 @@
pub mod data;
pub(crate) mod internal_data;
use super::*;
use crate::{
model::payload::{ClientDisconnect, Speaking},
tracks::{TrackHandle, TrackState},
};
pub use data as context_data;
use data::*;
use internal_data::*;
/// Information about which tracks or data fired an event.
///
/// [`Track`] events may be local or global, and have no tracks
/// if fired on the global context via [`Driver::add_global_event`].
///
/// [`Track`]: crate::tracks::Track
/// [`Driver::add_global_event`]: crate::driver::Driver::add_global_event
#[derive(Debug)]
#[non_exhaustive]
pub enum EventContext<'a> {
/// Track event context, passed to events created via [`TrackHandle::add_event`],
/// [`EventStore::add_event`], or relevant global events.
///
/// [`EventStore::add_event`]: EventStore::add_event
/// [`TrackHandle::add_event`]: TrackHandle::add_event
Track(&'a [(&'a TrackState, &'a TrackHandle)]),
/// Speaking state update, typically describing how another voice
/// user is transmitting audio data. Clients must send at least one such
/// packet to allow SSRC/UserID matching.
SpeakingStateUpdate(Speaking),
#[cfg(feature = "receive")]
/// Reordered and decoded audio packets, received every 20ms.
VoiceTick(VoiceTick),
#[cfg(feature = "receive")]
/// Opus audio packet, received from another stream.
RtpPacket(RtpData),
#[cfg(feature = "receive")]
/// Telemetry/statistics packet, received from another stream.
RtcpPacket(RtcpData),
/// Fired whenever a client disconnects.
ClientDisconnect(ClientDisconnect),
/// Fires when this driver successfully connects to a voice channel.
DriverConnect(ConnectData<'a>),
/// Fires when this driver successfully reconnects after a network error.
DriverReconnect(ConnectData<'a>),
/// Fires when this driver fails to connect to, or drops from, a voice channel.
DriverDisconnect(DisconnectData<'a>),
}
#[derive(Debug)]
pub enum CoreContext {
SpeakingStateUpdate(Speaking),
#[cfg(feature = "receive")]
VoiceTick(VoiceTick),
#[cfg(feature = "receive")]
RtpPacket(InternalRtpPacket),
#[cfg(feature = "receive")]
RtcpPacket(InternalRtcpPacket),
ClientDisconnect(ClientDisconnect),
DriverConnect(InternalConnect),
DriverReconnect(InternalConnect),
DriverDisconnect(InternalDisconnect),
}
impl<'a> CoreContext {
pub(crate) fn to_user_context(&'a self) -> EventContext<'a> {
match self {
Self::SpeakingStateUpdate(evt) => EventContext::SpeakingStateUpdate(*evt),
#[cfg(feature = "receive")]
Self::VoiceTick(evt) => EventContext::VoiceTick(evt.clone()),
#[cfg(feature = "receive")]
Self::RtpPacket(evt) => EventContext::RtpPacket(RtpData::from(evt)),
#[cfg(feature = "receive")]
Self::RtcpPacket(evt) => EventContext::RtcpPacket(RtcpData::from(evt)),
Self::ClientDisconnect(evt) => EventContext::ClientDisconnect(*evt),
Self::DriverConnect(evt) => EventContext::DriverConnect(ConnectData::from(evt)),
Self::DriverReconnect(evt) => EventContext::DriverReconnect(ConnectData::from(evt)),
Self::DriverDisconnect(evt) =>
EventContext::DriverDisconnect(DisconnectData::from(evt)),
}
}
}
impl EventContext<'_> {
/// Retreive the event class for an event (i.e., when matching)
/// an event against the registered listeners.
#[must_use]
pub fn to_core_event(&self) -> Option<CoreEvent> {
match self {
Self::SpeakingStateUpdate(_) => Some(CoreEvent::SpeakingStateUpdate),
#[cfg(feature = "receive")]
Self::VoiceTick(_) => Some(CoreEvent::VoiceTick),
#[cfg(feature = "receive")]
Self::RtpPacket(_) => Some(CoreEvent::RtpPacket),
#[cfg(feature = "receive")]
Self::RtcpPacket(_) => Some(CoreEvent::RtcpPacket),
Self::ClientDisconnect(_) => Some(CoreEvent::ClientDisconnect),
Self::DriverConnect(_) => Some(CoreEvent::DriverConnect),
Self::DriverReconnect(_) => Some(CoreEvent::DriverReconnect),
Self::DriverDisconnect(_) => Some(CoreEvent::DriverDisconnect),
_ => None,
}
}
}

77
app/vendor/songbird/src/events/core.rs vendored Normal file
View File

@@ -0,0 +1,77 @@
/// Voice core events occur on receipt of
/// voice packets and telemetry.
///
/// Core events persist while the `action` in [`EventData`]
/// returns `None`.
///
/// ## Events from other users
/// Songbird can observe when a user *speaks for the first time* ([`SpeakingStateUpdate`]),
/// when a client leaves the session ([`ClientDisconnect`]).
///
/// When the `"receive"` feature is enabled, songbird can also handle voice packets
#[cfg_attr(feature = "receive", doc = "([`RtpPacket`](Self::RtpPacket)),")]
#[cfg_attr(not(feature = "receive"), doc = "(`RtpPacket`),")]
/// decode and track speaking users
#[cfg_attr(feature = "receive", doc = "([`VoiceTick`](Self::VoiceTick)),")]
#[cfg_attr(not(feature = "receive"), doc = "(`VoiceTick`),")]
/// and handle telemetry data
#[cfg_attr(feature = "receive", doc = "([`RtcpPacket`](Self::RtcpPacket)).")]
#[cfg_attr(not(feature = "receive"), doc = "(`RtcpPacket`).")]
/// The format of voice packets is described by
#[cfg_attr(
feature = "receive",
doc = "[`VoiceData`](super::context::data::VoiceData)."
)]
#[cfg_attr(not(feature = "receive"), doc = "`VoiceData`.")]
///
/// To detect when a user connects, you must correlate gateway (e.g., `VoiceStateUpdate`) events
/// from the main part of your bot.
///
/// To obtain a user's SSRC, you must use [`SpeakingStateUpdate`] events.
///
/// [`EventData`]: super::EventData
/// [`SpeakingStateUpdate`]: Self::SpeakingStateUpdate
/// [`ClientDisconnect`]: Self::ClientDisconnect
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, enum_map::Enum)]
#[non_exhaustive]
pub enum CoreEvent {
/// Speaking state update from the WS gateway, typically describing how another voice
/// user is transmitting audio data. Clients must send at least one such
/// packet to allow SSRC/UserID matching.
///
/// Fired on receipt of a speaking state update from another host.
///
/// Note: this will fire when a user starts speaking for the first time,
/// or changes their capabilities.
SpeakingStateUpdate,
#[cfg(feature = "receive")]
/// Fires every 20ms, containing the scheduled voice packet and decoded audio
/// data for each live user.
VoiceTick,
#[cfg(feature = "receive")]
/// Fires on receipt of a voice packet from another stream in the voice call.
///
/// As RTP packets do not map to Discord's notion of users, SSRCs must be mapped
/// back using the user IDs seen through client connection, disconnection,
/// or speaking state update.
RtpPacket,
#[cfg(feature = "receive")]
/// Fires on receipt of an RTCP packet, containing various call stats
/// such as latency reports.
RtcpPacket,
/// Fires whenever a user disconnects from the same stream as the bot.
ClientDisconnect,
/// Fires when this driver successfully connects to a voice channel.
DriverConnect,
/// Fires when this driver successfully reconnects after a network error.
DriverReconnect,
/// Fires when this driver fails to connect to, or drops from, a voice channel.
DriverDisconnect,
}

82
app/vendor/songbird/src/events/data.rs vendored Normal file
View File

@@ -0,0 +1,82 @@
use super::*;
use std::cmp::Ordering;
/// Internal representation of an event, as handled by the audio context.
pub struct EventData {
pub(crate) event: Event,
pub(crate) fire_time: Option<Duration>,
pub(crate) action: Box<dyn EventHandler>,
}
impl EventData {
/// Create a representation of an event and its associated handler.
///
/// An event handler, `action`, receives an [`EventContext`] and optionally
/// produces a new [`Event`] type for itself. Returning `None` will
/// maintain the same event type, while removing any [`Delayed`] entries.
/// Event handlers will be re-added with their new trigger condition,
/// or removed if [`Cancel`]led
///
/// [`EventContext`]: EventContext
/// [`Event`]: Event
/// [`Delayed`]: Event::Delayed
/// [`Cancel`]: Event::Cancel
pub fn new<F: EventHandler + 'static>(event: Event, action: F) -> Self {
Self {
event,
fire_time: None,
action: Box::new(action),
}
}
/// Computes the next firing time for a timer event.
pub fn compute_activation(&mut self, now: Duration) {
match self.event {
Event::Periodic(period, phase) => {
self.fire_time = Some(now + phase.unwrap_or(period));
},
Event::Delayed(offset) => {
self.fire_time = Some(now + offset);
},
_ => {},
}
}
}
impl std::fmt::Debug for EventData {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
write!(
f,
"Event {{ event: {:?}, fire_time: {:?}, action: <fn> }}",
self.event, self.fire_time
)
}
}
/// Events are ordered/compared based on their firing time.
impl Ord for EventData {
fn cmp(&self, other: &Self) -> Ordering {
// FIXME: we don't have let chains in this edition songbird uses so when we upgrade to 2024 edition,
// change this to a let chain that way it reads easier
if let Some(t1) = &self.fire_time {
if let Some(t2) = &other.fire_time {
return t2.cmp(t1);
}
}
Ordering::Equal
}
}
impl PartialOrd for EventData {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other))
}
}
impl PartialEq for EventData {
fn eq(&self, other: &Self) -> bool {
self.fire_time == other.fire_time
}
}
impl Eq for EventData {}

147
app/vendor/songbird/src/events/mod.rs vendored Normal file
View File

@@ -0,0 +1,147 @@
//! Events relating to tracks, timing, and other callers.
//!
//! ## Listening for events
//! Driver events in Songbird are composed of two parts:
//! * An [`Event`] to listen out for. These may be discrete events,
//! or generated by timers.
//! * An [`EventHandler`] to be called on receipt of an event. As event
//! handlers may be shared between several events, the handler is called
//! with an [`EventContext`] describing which event was fired.
//!
//! Event handlers are registered using functions such as [`Driver::add_global_event`],
//! or [`TrackHandle::add_event`], or. Internally, these pairs are stored
//! as [`EventData`].
//!
//! ## `EventHandler` lifecycle
//! An event handler is essentially just an async function which may return
//! another type of event to listen out for (an `Option<Event>`). For instance,
//! [`Some(Event::Cancel)`] will remove that event listener, while `None` won't
//! change it at all.
//!
//! The exception is one-off events like [`Event::Delayed`], which remove themselves
//! after one call *unless* an [`Event`] override is returned.
//!
//! ## Global and local listeners
//! *Global* event listeners are those which are placed onto the [`Driver`],
//! while *local* event listeners are those which are placed on a
//! [`Track`]/[`Handle`].
//!
//! Track or timed events, when local, return a reference to the parent track.
//! When registered globally, they fire on a per-tick basis, returning references to
//! all relevant tracks in that 20ms window. Global/local timed events use a global
//! timer or a [track's playback time], respectively.
//!
//! [`CoreEvent`]s may only be registered globally.
//!
//! [`Event`]: Event
//! [`EventHandler`]: EventHandler
//! [`EventContext`]: EventContext
//! [`Driver::add_global_event`]: crate::driver::Driver::add_global_event
//! [`Driver`]: crate::driver::Driver::add_global_event
//! [`TrackHandle::add_event`]: crate::tracks::TrackHandle::add_event
//! [`Track`]: crate::tracks::Track::events
//! [`Handle`]: crate::tracks::TrackHandle::add_event
//! [`EventData`]: EventData
//! [`Some(Event::Cancel)`]: Event::Cancel
//! [`Event::Delayed`]: Event::Delayed
//! [track's playback time]: crate::tracks::TrackState::play_time
//! [`CoreEvent`]: CoreEvent
mod context;
mod core;
mod data;
mod store;
mod track;
mod untimed;
pub use self::{
context::{context_data, EventContext},
core::*,
data::*,
store::*,
track::*,
untimed::*,
};
pub(crate) use context::{internal_data, CoreContext};
use async_trait::async_trait;
use std::time::Duration;
/// Trait to handle an event which can be fired per-track, or globally.
///
/// These may be feasibly reused between several event sources.
#[async_trait]
pub trait EventHandler: Send + Sync {
/// Respond to one received event.
async fn act(&self, ctx: &EventContext<'_>) -> Option<Event>;
}
/// Classes of event which may occur, triggering a handler
/// at the local (track-specific) or global level.
///
/// Local time-based events rely upon the current playback
/// time of a track, and so will not fire if a track becomes paused
/// or stops. In case this is required, global events are a better
/// fit.
///
/// Event handlers themselves are described in [`EventData::new`].
///
/// [`EventData::new`]: EventData::new
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
#[non_exhaustive]
pub enum Event {
/// Periodic events rely upon two parameters: a *period*
/// and an optional *phase*.
///
/// If the *phase* is `None`, then the event will first fire
/// in one *period*. Periodic events repeat automatically
/// so long as the `action` in [`EventData`] returns `None`.
///
/// [`EventData`]: EventData
Periodic(Duration, Option<Duration>),
/// Delayed events rely upon a *delay* parameter, and
/// fire one *delay* after the audio context processes them.
///
/// Delayed events are automatically removed once fired,
/// so long as the `action` in [`EventData`] returns `None`.
///
/// [`EventData`]: EventData
Delayed(Duration),
/// Track events correspond to certain actions or changes
/// of state, such as a track finishing, looping, or being
/// manually stopped.
///
/// Track events persist while the `action` in [`EventData`]
/// returns `None`.
///
/// [`EventData`]: EventData
Track(TrackEvent),
/// Core events
///
/// Track events persist while the `action` in [`EventData`]
/// returns `None`. Core events **must** be applied globally,
/// as attaching them to a track is a no-op.
///
/// [`EventData`]: EventData
Core(CoreEvent),
/// Cancels the event, if it was intended to persist.
Cancel,
}
impl Event {
pub(crate) fn is_global_only(&self) -> bool {
matches!(self, Self::Core(_))
}
}
impl From<TrackEvent> for Event {
fn from(evt: TrackEvent) -> Self {
Event::Track(evt)
}
}
impl From<CoreEvent> for Event {
fn from(evt: CoreEvent) -> Self {
Event::Core(evt)
}
}

263
app/vendor/songbird/src/events/store.rs vendored Normal file
View File

@@ -0,0 +1,263 @@
use super::*;
use crate::{
constants::*,
tracks::{ReadyState, TrackHandle, TrackState},
};
use enum_map::EnumMap;
use std::collections::{BinaryHeap, HashMap};
use tracing::info;
#[derive(Debug, Default)]
/// Storage for [`EventData`], designed to be used for both local and global contexts.
///
/// Timed events are stored in a binary heap for fast selection, and have custom `Eq`,
/// `Ord`, etc. implementations to support (only) this.
///
/// [`EventData`]: EventData
pub struct EventStore {
timed: BinaryHeap<EventData>,
untimed: Box<EnumMap<UntimedEvent, Vec<EventData>>>,
local_only: bool,
}
impl EventStore {
/// Creates a new event store to be used globally.
#[must_use]
pub fn new() -> Self {
Self::default()
}
/// Creates a new event store to be used within a [`Track`].
///
/// This is usually automatically installed by the driver once
/// a track has been registered.
///
/// [`Track`]: crate::tracks::Track
#[must_use]
pub fn new_local() -> Self {
EventStore {
local_only: true,
..Default::default()
}
}
/// Add an event to this store.
///
/// Updates `evt` according to [`EventData::compute_activation`].
///
/// [`EventData::compute_activation`]: EventData::compute_activation
pub fn add_event(&mut self, mut evt: EventData, now: Duration) {
evt.compute_activation(now);
if self.local_only && evt.event.is_global_only() {
return;
}
match evt.event {
Event::Core(c) => {
self.untimed[c.into()].push(evt);
},
Event::Track(t) => {
self.untimed[t.into()].push(evt);
},
Event::Delayed(_) | Event::Periodic(_, _) => {
self.timed.push(evt);
},
_ => {
// Event cancelled.
},
}
}
/// Processes all events due up to and including `now`.
pub(crate) async fn process_timed(&mut self, now: Duration, ctx: EventContext<'_>) {
while let Some(evt) = self.timed.peek() {
if evt
.fire_time
.as_ref()
.expect("Timed event must have a fire_time.")
> &now
{
break;
}
let mut evt = self
.timed
.pop()
.expect("Can only succeed due to peek = Some(...).");
let old_evt_type = evt.event;
if let Some(new_evt_type) = evt.action.act(&ctx).await {
evt.event = new_evt_type;
self.add_event(evt, now);
} else if let Event::Periodic(d, _) = old_evt_type {
evt.event = Event::Periodic(d, None);
self.add_event(evt, now);
}
}
}
/// Processes all events due up to and including `now`.
pub(crate) fn timed_event_ready(&self, now: Duration) -> bool {
self.timed.peek().is_some_and(|evt| {
evt.fire_time
.as_ref()
.expect("Timed event must have a fire_time.")
<= &now
})
}
/// Processes all events attached to the given track event.
pub(crate) async fn process_untimed(
&mut self,
now: Duration,
untimed_event: UntimedEvent,
ctx: EventContext<'_>,
) {
// move a Vec in and out: not too expensive, but could be better.
// Although it's obvious that moving an event out of one vec and into
// another necessitates that they be different event types, thus entries,
// convincing the compiler of this is non-trivial without making them dedicated
// fields.
let mut events = std::mem::take(&mut self.untimed[untimed_event]);
if !events.is_empty() {
// TODO: Possibly use tombstones to prevent realloc/memcpys?
// i.e., never shrink array, replace ended tracks with <DEAD>,
// maintain a "first-track" stack and freelist alongside.
let mut i = 0;
while i < events.len() {
let evt = &mut events[i];
// Only remove/readd if the event type changes (i.e., Some AND new != old)
if let Some(new_evt_type) = evt.action.act(&ctx).await {
if evt.event == new_evt_type {
let mut evt = events.remove(i);
evt.event = new_evt_type;
self.add_event(evt, now);
} else {
i += 1;
}
} else {
i += 1;
};
}
self.untimed[untimed_event] = events;
}
}
}
#[derive(Debug, Default)]
pub(crate) struct GlobalEvents {
pub(crate) store: EventStore,
pub(crate) time: Duration,
pub(crate) awaiting_tick: HashMap<TrackEvent, Vec<usize>>,
}
impl GlobalEvents {
pub(crate) fn add_event(&mut self, evt: EventData) {
self.store.add_event(evt, self.time);
}
pub(crate) async fn fire_core_event(&mut self, evt: CoreEvent, ctx: EventContext<'_>) {
self.store.process_untimed(self.time, evt.into(), ctx).await;
}
pub(crate) fn fire_track_event(&mut self, evt: TrackEvent, index: usize) {
let holder = self.awaiting_tick.entry(evt).or_default();
holder.push(index);
}
pub(crate) fn remove_handlers(&mut self) {
self.store = EventStore::new();
}
pub(crate) async fn tick(
&mut self,
events: &mut [EventStore],
states: &mut [TrackState],
handles: &mut [TrackHandle],
) {
// Global timed events
self.time += TIMESTEP_LENGTH;
if self.store.timed_event_ready(self.time) {
let global_ctx: Vec<(&TrackState, &TrackHandle)> =
states.iter().zip(handles.iter()).collect();
self.store
.process_timed(self.time, EventContext::Track(&global_ctx[..]))
.await;
}
// Local timed events
for (i, state) in states.iter_mut().enumerate() {
if state.playing.is_playing() && state.ready == ReadyState::Playable {
state.step_frame();
let event_store = events
.get_mut(i)
.expect("Missing store index for Tick (local timed).");
let handle = handles
.get_mut(i)
.expect("Missing handle index for Tick (local timed).");
event_store
.process_timed(state.play_time, EventContext::Track(&[(state, handle)]))
.await;
}
}
for (evt, indices) in &self.awaiting_tick {
let untimed = (*evt).into();
if !indices.is_empty() {
info!("Firing {:?} for {:?}", evt, indices);
}
// Local untimed track events.
for &i in indices {
let event_store = events
.get_mut(i)
.expect("Missing store index for Tick (local untimed).");
let handle = handles
.get_mut(i)
.expect("Missing handle index for Tick (local untimed).");
let state = states
.get_mut(i)
.expect("Missing state index for Tick (local untimed).");
event_store
.process_untimed(
state.position,
untimed,
EventContext::Track(&[(state, handle)]),
)
.await;
}
// Global untimed track events.
if !self.store.untimed[untimed].is_empty() && !indices.is_empty() {
let global_ctx: Vec<(&TrackState, &TrackHandle)> = indices
.iter()
.map(|i| {
(
states
.get(*i)
.expect("Missing state index for Tick (global untimed)"),
handles
.get(*i)
.expect("Missing handle index for Tick (global untimed)"),
)
})
.collect();
self.store
.process_untimed(self.time, untimed, EventContext::Track(&global_ctx[..]))
.await;
}
}
// Now drain vecs.
for indices in self.awaiting_tick.values_mut() {
indices.clear();
}
}
}

35
app/vendor/songbird/src/events/track.rs vendored Normal file
View File

@@ -0,0 +1,35 @@
// TODO: Could this be a bitset? Could accelerate lookups,
// allow easy joint subscription & remove Vecs for related evt handling?
/// Track events correspond to certain actions or changes
/// of state, such as a track finishing, looping, or being
/// manually stopped. Voice core events occur on receipt of
/// voice packets and telemetry.
///
/// Track events persist while the `action` in [`EventData`]
/// returns `None`.
///
/// [`EventData`]: super::EventData
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, enum_map::Enum)]
#[non_exhaustive]
pub enum TrackEvent {
/// The attached track has resumed playing.
///
/// This event will not fire when a track first starts,
/// but will fire when a track changes from, e.g., paused to playing.
/// This is most relevant for queue users: queued tracks placed into a
/// non-empty queue are initlally paused, and are later moved to `Play`.
Play,
/// The attached track has been paused.
Pause,
/// The attached track has ended.
End,
/// The attached track has looped.
Loop,
/// The attached track is being readied or recreated.
Preparing,
/// The attached track has become playable.
Playable,
/// The attached track has encountered a runtime or initialisation error.
Error,
}

View File

@@ -0,0 +1,29 @@
use super::*;
/// Track and voice core events.
///
/// Untimed events persist while the `action` in [`EventData`]
/// returns `None`.
///
/// [`EventData`]: EventData
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, enum_map::Enum)]
#[non_exhaustive]
pub enum UntimedEvent {
/// Untimed events belonging to a track, such as state changes, end, or loops.
Track(TrackEvent),
/// Untimed events belonging to the global context, such as finished tracks,
/// client speaking updates, or RT(C)P voice and telemetry data.
Core(CoreEvent),
}
impl From<TrackEvent> for UntimedEvent {
fn from(evt: TrackEvent) -> Self {
UntimedEvent::Track(evt)
}
}
impl From<CoreEvent> for UntimedEvent {
fn from(evt: CoreEvent) -> Self {
UntimedEvent::Core(evt)
}
}

View File

@@ -0,0 +1,64 @@
use nonmax::NonMaxU32;
use std::time::Duration;
/// A type similar to [`std::time::Duration`] except using an `f32`.
///
/// This limits precision, only allowing by-second precision up to ~194 days,
/// but in turn shinks type size from `16` bytes to `4` bytes.
///
/// # Implementation Note
///
/// We actually store this as `NonMaxU32`, in order to get a niche to make `Option<FloatDuration>` the same size.
#[derive(Clone, Copy)]
pub struct FloatDuration(NonMaxU32);
impl FloatDuration {
/// Creates a [`FloatDuration`] from a raw `f32` value.
///
/// # Panics
/// This may panic if `value` is a NaN value.
#[must_use]
pub fn from_secs_f32(value: f32) -> Self {
Self(NonMaxU32::new(value.to_bits()).expect("value should not be NaN"))
}
/// Converts a [`FloatDuration`] to an `f32`.
///
/// This is cheaper than converting to a [`Duration`] and using [`Duration::as_secs_f32`] as it is more direct.
#[must_use]
pub fn as_secs_f32(self) -> f32 {
f32::from_bits(self.0.get())
}
}
impl std::fmt::Debug for FloatDuration {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_tuple("FloatDuration")
.field(&Duration::from(*self))
.finish()
}
}
impl PartialEq for FloatDuration {
fn eq(&self, other: &Self) -> bool {
self.as_secs_f32() == other.as_secs_f32()
}
}
impl PartialOrd for FloatDuration {
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
self.as_secs_f32().partial_cmp(&other.as_secs_f32())
}
}
impl From<Duration> for FloatDuration {
fn from(value: Duration) -> Self {
Self::from_secs_f32(value.as_secs_f32())
}
}
impl From<FloatDuration> for Duration {
fn from(value: FloatDuration) -> Self {
Duration::from_secs_f32(value.as_secs_f32())
}
}

Some files were not shown because too many files have changed in this diff Show More