From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 From: lucascouts Date: Sun, 18 Oct 2026 22:54:22 -0300 Subject: [PATCH] voice_input: dictate into the agent composer, transcribed on this machine Typing a long instruction to the agent is often slower than saying it. This adds a microphone to the agent composer: the speech is turned into text on this machine by whisper.cpp and put into the composer unsent, to be read and fixed before it is sent. Story 044. What it adds: * crate voice_input: the `voice_input` settings section, the pinned model table, the model download, microphone capture (cpal on its own thread, the device chosen by the audio crate's input setting; F32, I16, U16, I24 and I32 inputs, 120 s at most), resampling to 16 kHz mono with rodio, local (whisper-rs) and remote transcription, and a pure state machine (state.rs) that decides what a click, a key press and a key release mean. One app-wide VoiceInput entity runs its effects. Inputs and events carry the composer's entity id, so one recording runs at a time; another composer is told "Voice input is busy in another thread". * agent_ui: a mic button beside Send (m:ss while recording, a spinner while transcribing, disabled when read-only), a strip above the editor for the download consent and its progress, and MessageEditor::insert_transcript: it replaces the newest selection as one undo step, adds a space when the char before the selection's start is not whitespace, and never sends. A transcript for a closed thread is discarded with a toast; one refused by a read-only composer is logged outcome=refused. * Keys, in all four default keymaps (vim.json included): voice_input::ToggleRecording on ctrl-alt-v (cmd-alt-v on macOS). A tap (< 300 ms) toggles, a hold is push-to-talk until the key or its modifiers are released, and the command palette's toggle is always a tap. Escape cancels a recording or declines the consent, except while a completion menu is open, which it closes first. * reqwest_client: ReqwestClient::without_proxy, which zed/main.rs hands to voice_input for plain-http loopback URLs. * zlog: whisper_rs is filtered at warn (it logs model details at info on every load). Why a patch: an extension can neither record audio nor add a button to the composer. Why these two crates: nothing in the tree turns speech into text. whisper-rs 0.16.0 (Unlicense) is a safe API over whisper.cpp; whisper-rs-sys 0.15.0 (Unlicense) builds whisper.cpp 1.8.3 (MIT) through CMake from its bundled source. Both were the newest on crates.io on 2026-10-18, published 2026-03-12. They are the only third-party crates Cargo.lock gains, beside voice_input itself. Resampling (rodio), capture (cpal) and hashing (sha2) use crates already locked; the WAV for the remote service is 44 bytes by hand. The model: /usr/share/whisper-models (app-accessibility/whisper-models) first, its size checked on every use and its SHA-256 once per path and mtime; else /voice/models. A missing model is downloaded only after a click on Download in the consent strip, from Hugging Face ggerganov/whisper.cpp at commit 5359861c739e955e79d9a303bcbc70fb988958b1: https://huggingface.co/ggerganov/whisper.cpp/resolve//ggml-.bin tiny 77691713 B be07e048e1e599ad46341c8d2a135645097a538221678b7acdd1b1919c6e1b21 base 147951465 B 60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe small 487601967 B 1be3a9b2063867b937e64e2ec7483364a79917e157fa98c5d94b5c1fffea987b It is written to a .download-* file in the same directory, held under an exclusive lock, and renamed into place only with the pinned size and SHA-256. A cancel, a 30 s stall, an oversized body or a wrong digest leaves nothing at the final path. A download outlives the composer that asked for it, and the next composer adopts its strip. Stale .download-* files (older than 60 s, held by no running Zeo) are swept at start. A downloaded model whisper cannot load is deleted, to be downloaded again; a packaged one is left alone. Privacy: local mode opens no connection; the model download, after consent, is the only network use. The remote service is off by default (voice_input.remote.enabled: false). When it is on, the recording goes as a 16 kHz mono WAV to the OpenAI-compatible endpoint in voice_input.remote.url: https, or plain http to a loopback host, sent direct past any proxy. The mic button's tooltip names the host. The API key is read per request from the environment variable that voice_input.remote.api_key_env names (default ZEO_VOICE_API_KEY), sent only as a bearer token, never written to settings and never logged. Neither mode falls back to the other. Logs are key=value lines ("voice_input event=recording mode= audio_ms= backend= model= transcribe_ms= outcome=", event=model_download, ...) with no transcript, audio, key, or URL user-info and query. Settings ("voice_input" in settings.json): enabled (true), model ("base"; "tiny" or "small"), language ("auto" or a two-letter code), remote { enabled (false), url (null), model ("whisper-1"), api_key_env ("ZEO_VOICE_API_KEY") }. "enabled": false hides the button and leaves the shortcut unbound. Packaging: USE voice (default on) in app-editors/zeo, its own series group after every devtools group and epic, and before test. Generated on top of the claude-agent-acp-plus group, whose composer hunks (0043-0045) it sits beside, so REQUIRED_USE ties voice? to it. It needs none of the optional groups: it applies with no fuzz with claude-code-ide, devtools-bridge, ai-jail, browser-extension, devtools-panel and epic each on or off (eleven combinations checked), the zed crate and the voice tests build with only the acp-plus group under them, and 0050/0051 apply on top of it unchanged. The ebuild adds whisper-rs@0.16.0 and whisper-rs-sys@0.15.0 to CRATES, LICENSE voice? ( Unlicense ), and exports GGML_NATIVE=OFF and WHISPER_DONT_GENERATE_BINDINGS=1 (with ggml's own x86 flags off), so whisper.cpp follows CFLAGS and builds from its bundled bindings. Tests: cargo test -p voice_input (204: state machine, transcript, audio, models, settings, download, capture, local and remote, entity), cargo test -p agent_ui voice (64), cargo test -p reqwest_client without_proxy. --- diff --git a/Cargo.lock b/Cargo.lock index 9dfc504..be6ec7d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -546,6 +546,7 @@ dependencies = [ "command_palette_hooks", "component", "context_server", + "cpal", "db", "editor", "eval_utils", @@ -623,6 +624,7 @@ dependencies = [ "util", "uuid", "vim", + "voice_input", "watch", "workspace", "zed_actions", @@ -2304,6 +2306,8 @@ dependencies = [ "cexpr", "clang-sys", "itertools 0.13.0", + "log", + "prettyplease", "proc-macro2", "quote", "regex", @@ -19835,6 +19839,30 @@ dependencies = [ "settings", ] +[[package]] +name = "voice_input" +version = "0.1.0" +dependencies = [ + "anyhow", + "audio", + "cpal", + "futures 0.3.34", + "gpui", + "http_client", + "libc", + "log", + "paths", + "rodio", + "serde", + "serde_json", + "settings", + "sha2", + "smol", + "tempfile", + "url", + "whisper-rs", +] + [[package]] name = "vscode_theme" version = "0.2.0" @@ -20913,6 +20941,29 @@ dependencies = [ "zed_actions", ] +[[package]] +name = "whisper-rs" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2088172d00f936c348d6a72f488dc2660ab3f507263a195df308a3c2383229f6" +dependencies = [ + "log", + "whisper-rs-sys", +] + +[[package]] +name = "whisper-rs-sys" +version = "0.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6986c0fe081241d391f09b9a071fbcbb59720c3563628c3c829057cf69f2a56f" +dependencies = [ + "bindgen", + "cfg-if", + "cmake", + "fs_extra", + "semver", +] + [[package]] name = "whoami" version = "1.6.1" @@ -22754,6 +22805,7 @@ dependencies = [ "uuid", "vim", "vim_mode_setting", + "voice_input", "watch", "web_search", "web_search_providers", diff --git a/Cargo.toml b/Cargo.toml index 8a9bf11..2f0a724 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -240,6 +240,7 @@ members = [ "crates/util_macros", "crates/vim", "crates/vim_mode_setting", + "crates/voice_input", "crates/watch", "crates/web_search", "crates/web_search_providers", @@ -520,6 +521,7 @@ util = { path = "crates/util" } util_macros = { path = "crates/util_macros" } vim = { path = "crates/vim" } vim_mode_setting = { path = "crates/vim_mode_setting" } +voice_input = { path = "crates/voice_input" } watch = { path = "crates/watch" } web_search = { path = "crates/web_search" } web_search_providers = { path = "crates/web_search_providers" } @@ -1019,6 +1021,7 @@ webrtc-sys = "0.3.23" # 30.0.1 lacks the Vulkan surface-drop fix (wgpu#10230) and WebGL resize fix (wgpu#10348). wgpu = { git = "https://github.com/gfx-rs/wgpu", rev = "cddb2f5d80dc0009b2ef6767635140cacf90a007", default-features = false } which = "8" +whisper-rs = { version = "0.16.0", default-features = false, features = ["log_backend"] } windows-core = "0.62" windows-registry = "0.6.0" xkbcommon = { version = "0.8", default-features = false } diff --git a/assets/keymaps/default-linux.json b/assets/keymaps/default-linux.json index eb5bcb5..8491fc1 100644 --- a/assets/keymaps/default-linux.json +++ b/assets/keymaps/default-linux.json @@ -401,6 +401,46 @@ "ctrl-enter": "agent::IgnoreSendWhileQuestionPending", }, }, + { + // Voice input: press and release to toggle recording, hold to talk. + "context": "AcpThread", + "bindings": { + "ctrl-alt-v": "voice_input::ToggleRecording", + }, + }, + { + // Live only while this composer records, so Escape keeps its usual + // meaning the rest of the time. + // Not while a completion menu shows: the editor's own Escape closes it + // first, and the recording goes on. + "context": "AcpThread > Editor && voice_recording && !menu", + "bindings": { + "escape": "voice_input::CancelRecording", + }, + }, + { + // Live only while this composer asks for the speech model download: + // declines it. Not while the download runs, which takes minutes: Escape + // keeps its usual meaning then, and the strip's Cancel button cancels. + "context": "AcpThread > Editor && voice_consent && !menu", + "bindings": { + "escape": "voice_input::DismissDownload", + }, + }, + { + // The same two, with focus elsewhere in the thread: on the strip's + // buttons, or in the conversation. + "context": "AcpThread && voice_recording", + "bindings": { + "escape": "voice_input::CancelRecording", + }, + }, + { + "context": "AcpThread && voice_consent", + "bindings": { + "escape": "voice_input::DismissDownload", + }, + }, { "context": "Elicitation || (Elicitation > Editor)", "use_key_equivalents": true, diff --git a/assets/keymaps/default-macos.json b/assets/keymaps/default-macos.json index 0bdcbdf..1c29405 100644 --- a/assets/keymaps/default-macos.json +++ b/assets/keymaps/default-macos.json @@ -435,6 +435,46 @@ "cmd-enter": "agent::IgnoreSendWhileQuestionPending", }, }, + { + // Voice input: press and release to toggle recording, hold to talk. + "context": "AcpThread", + "bindings": { + "cmd-alt-v": "voice_input::ToggleRecording", + }, + }, + { + // Live only while this composer records, so Escape keeps its usual + // meaning the rest of the time. + // Not while a completion menu shows: the editor's own Escape closes it + // first, and the recording goes on. + "context": "AcpThread > Editor && voice_recording && !menu", + "bindings": { + "escape": "voice_input::CancelRecording", + }, + }, + { + // Live only while this composer asks for the speech model download: + // declines it. Not while the download runs, which takes minutes: Escape + // keeps its usual meaning then, and the strip's Cancel button cancels. + "context": "AcpThread > Editor && voice_consent && !menu", + "bindings": { + "escape": "voice_input::DismissDownload", + }, + }, + { + // The same two, with focus elsewhere in the thread: on the strip's + // buttons, or in the conversation. + "context": "AcpThread && voice_recording", + "bindings": { + "escape": "voice_input::CancelRecording", + }, + }, + { + "context": "AcpThread && voice_consent", + "bindings": { + "escape": "voice_input::DismissDownload", + }, + }, { "context": "Elicitation || (Elicitation > Editor)", "use_key_equivalents": true, diff --git a/assets/keymaps/default-windows.json b/assets/keymaps/default-windows.json index a9f06d4..4d87b09 100644 --- a/assets/keymaps/default-windows.json +++ b/assets/keymaps/default-windows.json @@ -404,6 +404,46 @@ "ctrl-enter": "agent::IgnoreSendWhileQuestionPending", }, }, + { + // Voice input: press and release to toggle recording, hold to talk. + "context": "AcpThread", + "bindings": { + "ctrl-alt-v": "voice_input::ToggleRecording", + }, + }, + { + // Live only while this composer records, so Escape keeps its usual + // meaning the rest of the time. + // Not while a completion menu shows: the editor's own Escape closes it + // first, and the recording goes on. + "context": "AcpThread > Editor && voice_recording && !menu", + "bindings": { + "escape": "voice_input::CancelRecording", + }, + }, + { + // Live only while this composer asks for the speech model download: + // declines it. Not while the download runs, which takes minutes: Escape + // keeps its usual meaning then, and the strip's Cancel button cancels. + "context": "AcpThread > Editor && voice_consent && !menu", + "bindings": { + "escape": "voice_input::DismissDownload", + }, + }, + { + // The same two, with focus elsewhere in the thread: on the strip's + // buttons, or in the conversation. + "context": "AcpThread && voice_recording", + "bindings": { + "escape": "voice_input::CancelRecording", + }, + }, + { + "context": "AcpThread && voice_consent", + "bindings": { + "escape": "voice_input::DismissDownload", + }, + }, { "context": "Elicitation || (Elicitation > Editor)", "use_key_equivalents": true, diff --git a/assets/keymaps/vim.json b/assets/keymaps/vim.json index eca5695..ecac6c9 100644 --- a/assets/keymaps/vim.json +++ b/assets/keymaps/vim.json @@ -1229,6 +1229,50 @@ "enter": "agent::IgnoreSendWhileQuestionPending", }, }, + { + // Voice input. vim.json loads after the default keymaps, so vim's own + // Escape bindings (insert, normal and visual mode) would otherwise shadow + // cancelling a recording or declining the model download. The toggle + // follows the platform's default keymap: cmd-alt-v on macOS, ctrl-alt-v + // elsewhere. + "context": "AcpThread && os == macos", + "bindings": { + "cmd-alt-v": "voice_input::ToggleRecording", + }, + }, + { + "context": "AcpThread && os != macos", + "bindings": { + "ctrl-alt-v": "voice_input::ToggleRecording", + }, + }, + { + // An open completion menu takes the first Escape, as in the default keymaps. + "context": "AcpThread > Editor && voice_recording && !menu", + "bindings": { + "escape": "voice_input::CancelRecording", + }, + }, + { + "context": "AcpThread > Editor && voice_consent && !menu", + "bindings": { + "escape": "voice_input::DismissDownload", + }, + }, + { + // The same two, with focus elsewhere in the thread: on the strip's + // buttons, or in the conversation. + "context": "AcpThread && voice_recording", + "bindings": { + "escape": "voice_input::CancelRecording", + }, + }, + { + "context": "AcpThread && voice_consent", + "bindings": { + "escape": "voice_input::DismissDownload", + }, + }, { "context": "SettingsWindow > NavigationMenu && !search", diff --git a/assets/settings/default.json b/assets/settings/default.json index d546e09..3ab2e8f 100644 --- a/assets/settings/default.json +++ b/assets/settings/default.json @@ -3120,6 +3120,25 @@ // Example: {"log": {"client": "warn"}} "log": {}, + // Dictation into the agent composer. + "voice_input": { + // Whether the composer shows the microphone button and binds the voice shortcut. + "enabled": true, + // The local Whisper model: "tiny", "base" or "small". + "model": "base", + // "auto" lets the model detect the spoken language; a two-letter lowercase + // code such as "en" or "pt" fixes it. + "language": "auto", + // An OpenAI-compatible transcription service used instead of the local model. + // The API key is read from the environment variable `api_key_env` names. + "remote": { + "enabled": false, + "url": null, + "model": "whisper-1", + "api_key_env": "ZEO_VOICE_API_KEY", + }, + }, + // Configuration for developer-oriented instrumentation tools that can be // toggled at runtime. "instrumentation": { diff --git a/crates/agent_ui/Cargo.toml b/crates/agent_ui/Cargo.toml index 66b9f67..a19c044 100644 --- a/crates/agent_ui/Cargo.toml +++ b/crates/agent_ui/Cargo.toml @@ -115,6 +115,7 @@ unicode-segmentation.workspace = true url.workspace = true util.workspace = true uuid.workspace = true +voice_input.workspace = true watch.workspace = true workspace.workspace = true git_ui_core.workspace = true @@ -129,6 +130,7 @@ agent = { workspace = true, features = ["test-support"] } agent_servers = { workspace = true, features = ["test-support"] } buffer_diff = { workspace = true, features = ["test-support"] } client = { workspace = true, features = ["test-support"] } +cpal.workspace = true db = { workspace = true, features = ["test-support"] } editor = { workspace = true, features = ["test-support"] } eval_utils.workspace = true diff --git a/crates/agent_ui/src/agent_ui.rs b/crates/agent_ui/src/agent_ui.rs index c759ea0..424acd5 100644 --- a/crates/agent_ui/src/agent_ui.rs +++ b/crates/agent_ui/src/agent_ui.rs @@ -58,6 +58,8 @@ pub mod thread_worktree_archive; pub mod threads_archive_view; mod ui; mod unicode_confusables; +mod voice_button; +mod voice_notice; use std::rc::Rc; use std::sync::Arc; @@ -783,6 +785,7 @@ pub fn init( init_language_model_settings(cx); } agent_panel::init(cx); + voice_notice::init(cx); context_server_configuration::init(language_registry, fs.clone(), cx); thread_metadata_store::init(cx); thread_organization::init(cx); @@ -1136,6 +1139,43 @@ pub(crate) fn slash_command_in_code_block(code: &str) -> Option<&str> { Some(command) } +#[cfg(test)] +mod voice_insert_more_tests; +#[cfg(test)] +mod voice_insert_tests; + +#[cfg(test)] +mod voice_button_tests; + +#[cfg(test)] +mod voice_button_more_tests; + +#[cfg(test)] +mod voice_keys_tests; + +#[cfg(test)] +mod voice_keys_fix_tests; + +#[cfg(test)] +mod voice_refused_log_tests; + +#[cfg(test)] +mod voice_contract_tests; + +#[cfg(test)] +mod voice_download_adopt_ui_tests; + +#[cfg(test)] +mod voice_keymap_v3_tests; + +#[cfg(test)] +mod voice_keyless_tests; + +#[cfg(test)] +mod voice_events_more_tests; +#[cfg(test)] +mod voice_events_tests; + #[cfg(test)] mod slash_command_in_code_block_tests; diff --git a/crates/agent_ui/src/conversation_view.rs b/crates/agent_ui/src/conversation_view.rs index 4908f95..f1b829b 100644 --- a/crates/agent_ui/src/conversation_view.rs +++ b/crates/agent_ui/src/conversation_view.rs @@ -115,7 +115,7 @@ mod session_stats; mod sub_thread_header; mod subagent_card; mod thread_search_bar; -mod thread_view; +pub(crate) mod thread_view; pub use message_queue::*; pub use thread_view::*; diff --git a/crates/agent_ui/src/conversation_view/thread_view.rs b/crates/agent_ui/src/conversation_view/thread_view.rs index 495096b..3fc2d83 100644 --- a/crates/agent_ui/src/conversation_view/thread_view.rs +++ b/crates/agent_ui/src/conversation_view/thread_view.rs @@ -27,12 +27,22 @@ use sandbox::{SandboxFsPolicy, SandboxNetPolicy, SandboxPolicy}; use crate::completion_provider::{AvailableSkill, PromptLocalCommand, pluralize}; use crate::message_editor::SharedSessionCapabilities; +use crate::message_editor::VoiceKeyContext; use crate::set_as_default::{SET_AS_DEFAULT_HINT, default_marker, persists_default}; use crate::ui::{ SandboxGroup, SandboxRow, SandboxSection, SandboxStatusTooltip, TerminalSandboxWarning, TerminalToolHeader, }; use crate::unicode_confusables; +use crate::voice_button::{ + releases_binding, render_voice_button, render_voice_strip, send as send_voice_input, + toggle_bindings, voice_owner, +}; +use crate::voice_notice; +use voice_input::state::Input as VoiceCommand; +use voice_input::{ + CancelRecording, DismissDownload, ToggleRecording, ViewState, VoiceEvent, VoiceInput, +}; use db::kvp::KeyValueStore; use gpui::List; @@ -844,7 +854,27 @@ pub struct ThreadView { dismissed_skill_loading_issues: HashSet, pub(crate) thread_search_bar: Option>, pub(crate) thread_search_visible: bool, + voice_shortcut: VoiceShortcut, +} + +/// What this view knows of the voice shortcut that a key press left held. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +enum VoiceShortcut { + #[default] + Released, + /// `ToggleRecording` ran; the keystroke observer, which runs right after the action, says + /// which chord pressed it. + Pressed, + /// Held with these modifiers: letting go of every one of them is a release too. + Held(gpui::Modifiers), + /// Released by letting go of these modifiers while the key may still be down. Wayland + /// repeats a held key by replaying the keystroke of its press, modifiers included, until the + /// key comes up, so a press now is a stale repeat. It ends with the key-up of a binding's key, + /// a window deactivation, or one of these modifiers pressed again (macOS sends no key-up + /// while Cmd is held, so a modifier is the first sign of a new press there). + ReleasedByModifiers(gpui::Modifiers), } + impl Focusable for ThreadView { fn focus_handle(&self, _cx: &App) -> FocusHandle { self.focus_handle.clone() @@ -1378,12 +1408,14 @@ impl ThreadView { dismissed_skill_loading_issues: HashSet::default(), thread_search_bar: None, thread_search_visible: false, + voice_shortcut: VoiceShortcut::default(), }; this.sync_reported_activity(cx); this.sync_generating_indicator(cx); this.sync_editor_mode(cx); this.sync_existing_elicitation_states(window, cx); + this.subscribe_to_voice_input(window, cx); let list_state_for_scroll = this.list_state.clone(); let thread_view = cx.entity().downgrade(); @@ -6164,6 +6196,25 @@ impl ThreadView { let has_messages = self.has_visible_conversation(cx); let fills_container = !has_messages || editor_expanded; let composer_is_read_only = self.message_editor.read(cx).editor().read(cx).read_only(cx); + // `view_state` answers `Hidden` while voice input is disabled, and nothing renders then. + let voice = voice_input::VoiceInput::try_global(cx).map(|voice| { + let voice = voice.read(cx); + (voice.view_state(cx.entity_id()), voice.remote_host()) + }); + let voice_button = voice.as_ref().and_then(|(state, remote_host)| { + render_voice_button( + state, + composer_is_read_only, + cx.entity_id(), + remote_host.as_deref(), + &self.message_editor.focus_handle(cx), + window, + cx, + ) + }); + let voice_strip = voice + .as_ref() + .and_then(|(state, _)| render_voice_strip(state, cx)); let send_to_parent = self.render_send_to_parent(window, cx); let discard_draft_button = composer_is_read_only.then(|| { div() @@ -6177,6 +6228,7 @@ impl ThreadView { editor.set_read_only(false, cx); editor.set_message(Vec::new(), window, cx); }); + this.sync_voice_read_only(cx); this.clear_thread_error(cx); })), ) @@ -6210,6 +6262,7 @@ impl ThreadView { .flex_grow_0() .justify_between() .gap_2() + .children(voice_strip) .children(send_to_parent) .child( v_flex() @@ -6289,6 +6342,7 @@ impl ThreadView { .children(self.mode_selector.clone()) .children(self.model_selector.clone()), }) + .children(voice_button) .child(self.render_send_button(cx)), ), ), @@ -9463,6 +9517,259 @@ impl ThreadView { self.message_editor.read(cx).set_local_commands(commands); } + /// Makes this view a voice-input owner: a transcript finishing after it closed is discarded + /// (R4.7), and losing the window's focus ends a held recording (R3.3). + fn subscribe_to_voice_input(&mut self, window: &mut Window, cx: &mut Context) { + let Some(voice) = VoiceInput::try_global(cx) else { + return; + }; + let owner = voice_owner(cx.entity_id()); + let alive = cx.weak_entity().into(); + voice.update(cx, |voice, _| voice.register_owner(owner, alive)); + self.sync_voice_read_only(cx); + self._subscriptions.push(cx.subscribe_in( + &voice, + window, + |this, _, event: &VoiceEvent, window, cx| match event { + VoiceEvent::Changed => cx.notify(), + VoiceEvent::Insert { owner, text } if *owner == cx.entity_id() => { + this.insert_voice_transcript(text, window, cx) + } + VoiceEvent::Notify { owner, message } if *owner == cx.entity_id() => { + voice_notice::show_toast(&this.workspace, message.clone(), cx) + } + // Another composer's event. A discarded transcript has no view left to show it, + // so `agent_ui::init` tells the active workspace (R4.7). + VoiceEvent::Insert { .. } + | VoiceEvent::Notify { .. } + | VoiceEvent::Discarded { .. } => {} + }, + )); + self._subscriptions.push( + cx.observe_window_activation(window, move |this, window, cx| { + if !window.is_window_active() { + this.voice_shortcut = VoiceShortcut::Released; + send_voice_input(VoiceCommand::WindowDeactivated { owner }, cx); + } + }), + ); + // An action handler is not told which keystroke dispatched it, and the window's own + // modifiers are not a reliable record of the chord: the observer gets the keystroke. + self._subscriptions + .push(cx.observe_keystrokes(|this, event, window, cx| { + if this.voice_shortcut != VoiceShortcut::Pressed { + return; + } + // A keystroke bound to another action did not press the toggle: a dispatch nested + // in that action's handler stays `Pressed`, and the deferred check in + // `toggle_voice_recording` releases it with a `ShortcutUp`, as a tap. + if !event + .action + .as_ref() + .is_some_and(|action| action.as_any().is::()) + { + return; + } + let modifiers = event.keystroke.modifiers; + // The observer runs for every keystroke in every window: a view takes only a + // chord pressed where it has focus. (A press without a key no longer stays + // `Pressed`: `toggle_voice_recording` releases it once the dispatch is over.) + let pressed_by_a_chord = + modifiers.modified() && this.focus_handle.contains_focused(window, cx); + // Bound to a key with no modifiers: only the key-up can release it. + this.voice_shortcut = if pressed_by_a_chord { + VoiceShortcut::Held(modifiers) + } else { + VoiceShortcut::Released + }; + })); + } + + /// Puts a transcript into the composer (R4.1); a refusal, such as a read-only composer, is + /// shown rather than swallowed. + fn insert_voice_transcript(&mut self, text: &str, window: &mut Window, cx: &mut Context) { + let inserted = self + .message_editor + .update(cx, |editor, cx| editor.insert_transcript(text, window, cx)); + if let Err(error) = inserted { + voice_notice::show_toast(&self.workspace, format!("{error:#}"), cx); + // The recording's log line says `refused`, not `inserted` (R10.1); it names no text. + if let Some(voice) = VoiceInput::try_global(cx) { + let owner = voice_owner(cx.entity_id()); + voice.update(cx, |voice, _| voice.report_insert_refused(owner)); + } + } + } + + /// Tells voice input whether this composer is read-only: it refuses a start from a read-only + /// composer (R1.3) and knows only what this call tells it. + fn sync_voice_read_only(&self, cx: &mut Context) { + let Some(voice) = VoiceInput::try_global(cx) else { + return; + }; + let owner = voice_owner(cx.entity_id()); + let read_only = self.message_editor.read(cx).editor().read(cx).read_only(cx); + voice.update(cx, |voice, _| { + voice.set_composer_read_only(owner, read_only) + }); + } + + fn voice_enabled(cx: &App) -> bool { + VoiceInput::try_global(cx).is_some_and(|voice| voice.read(cx).is_enabled()) + } + + fn voice_view_state(&self, cx: &Context) -> Option { + VoiceInput::try_global(cx).map(|voice| voice.read(cx).view_state(cx.entity_id())) + } + + /// Names this composer's voice state in its key context, so Escape is claimed only while + /// there is a recording to cancel (R2.5) or a download consent to decline. The thread's own key + /// context names it too: focus can be elsewhere in the thread — on the strip's buttons, or + /// in the conversation — while the recording or the download is this thread's. + fn sync_voice_key_context(&self, cx: &Context) -> VoiceKeyContext { + let context = match self.voice_view_state(cx) { + Some(ViewState::Recording { .. }) => VoiceKeyContext::Recording, + Some(ViewState::AwaitingConsent { .. }) => VoiceKeyContext::Consent, + // The download runs in the background for minutes while the user keeps typing, so + // Escape keeps its usual meaning; the strip's Cancel button cancels it. + Some( + ViewState::Hidden + | ViewState::Idle { .. } + | ViewState::Busy + | ViewState::Downloading { .. } + | ViewState::Transcribing, + ) + | None => VoiceKeyContext::None, + }; + self.message_editor.read(cx).set_voice_key_context(context); + context + } + + fn thread_key_context(voice: VoiceKeyContext, os: gpui::OperatingSystem) -> gpui::KeyContext { + // Built with the defaults so it carries `os`: vim.json binds the voice toggle under + // `AcpThread && os == macos` / `os != macos`. + let mut context = gpui::KeyContext::new_with_defaults(os); + context.add("AcpThread"); + match voice { + VoiceKeyContext::None => {} + VoiceKeyContext::Recording => context.add("voice_recording"), + VoiceKeyContext::Consent => context.add("voice_consent"), + } + context + } + + /// A press of the voice shortcut. X11 repeats it while held; the state machine, not this + /// handler, tells a repeat from a fresh press (R3.2). After a release through the modifiers + /// the machine has let go of the key, so the view drops the repeats itself. + /// + /// Run without a keystroke (the command palette), it is a tap: no key-up will ever come, so + /// the press is released straight away (R2.7). + fn toggle_voice_recording( + &mut self, + _: &ToggleRecording, + window: &mut Window, + cx: &mut Context, + ) { + if let VoiceShortcut::ReleasedByModifiers(_) = self.voice_shortcut { + return; + } + self.sync_voice_read_only(cx); + self.voice_shortcut = VoiceShortcut::Pressed; + let owner = voice_owner(cx.entity_id()); + send_voice_input(VoiceCommand::ShortcutDown { owner }, cx); + // The keystroke observer runs synchronously right after a keyed action, so by the time + // this deferred check runs, a keyed press has left `Pressed`. Still `Pressed` means no + // keystroke claimed the action. + cx.defer_in(window, move |this, _window, cx| { + if this.voice_shortcut == VoiceShortcut::Pressed { + this.voice_shortcut = VoiceShortcut::Released; + send_voice_input(VoiceCommand::ShortcutUp { owner }, cx); + } + }); + } + + /// The release of the voice shortcut: a hold stops there (R3.1), a tap keeps recording + /// (R2.2). Key-ups reach only the focused path; window deactivation covers the rest (R3.3). + /// Any binding of the shortcut counts, not only the one the tooltip shows. + fn voice_key_up( + &mut self, + event: &gpui::KeyUpEvent, + window: &mut Window, + cx: &mut Context, + ) { + let focus = self.message_editor.focus_handle(cx); + if toggle_bindings(window, &focus) + .iter() + .any(|binding| releases_binding(&event.keystroke, binding)) + { + self.voice_shortcut = VoiceShortcut::Released; + let owner = voice_owner(cx.entity_id()); + send_voice_input(VoiceCommand::ShortcutUp { owner }, cx); + } + } + + /// Letting go of the held chord's modifiers is a release too. On macOS it is the only one + /// when `v` comes up before Cmd: AppKit delivers no key-up while Cmd is down. The key-up above + /// stays a release of its own, for a chord with no modifiers and for a release in the usual + /// order. When both arrive, the second finds the machine already released, which ignores it. + fn voice_modifiers_changed( + &mut self, + event: &gpui::ModifiersChangedEvent, + _window: &mut Window, + cx: &mut Context, + ) { + let pressed = match self.voice_shortcut { + VoiceShortcut::Held(pressed) => pressed, + VoiceShortcut::ReleasedByModifiers(released) => { + if (released & event.modifiers).modified() { + self.voice_shortcut = VoiceShortcut::Released; + } + return; + } + VoiceShortcut::Released | VoiceShortcut::Pressed => return, + }; + if (pressed & event.modifiers).modified() { + return; + } + self.voice_shortcut = VoiceShortcut::ReleasedByModifiers(pressed); + let owner = voice_owner(cx.entity_id()); + send_voice_input(VoiceCommand::ShortcutUp { owner }, cx); + } + + /// Escape while recording: the audio is discarded, nothing is transcribed (R2.5). + fn cancel_voice_recording( + &mut self, + _: &CancelRecording, + _window: &mut Window, + cx: &mut Context, + ) { + match self.voice_view_state(cx) { + Some(ViewState::Recording { .. }) => { + let owner = voice_owner(cx.entity_id()); + send_voice_input(VoiceCommand::Cancel { owner }, cx); + } + // The key context was stale: Escape keeps its usual meaning. + _ => cx.propagate(), + } + } + + /// Escape while the strip asks for the model download: declines it. A download already + /// running is cancelled only by the strip's Cancel button. + fn dismiss_voice_download( + &mut self, + _: &DismissDownload, + _window: &mut Window, + cx: &mut Context, + ) { + match self.voice_view_state(cx) { + Some(ViewState::AwaitingConsent { .. }) => { + send_voice_input(VoiceCommand::ConsentDeclined, cx) + } + // The key context was stale: Escape keeps its usual meaning. + _ => cx.propagate(), + } + } + /// Mirrors the conversation's pending question into the composer's key /// context. Read on every render rather than on the elicitation events, so /// every way a question ends -- answered, declined, cancelled, the turn @@ -15433,6 +15740,8 @@ impl Render for ThreadView { // renders (settings, connection state, feature flags). self.sync_local_commands(cx); self.sync_question_pending(cx); + let voice_key_context = self.sync_voice_key_context(cx); + let voice_enabled = Self::voice_enabled(cx); let has_messages = self.has_visible_conversation(cx); let list_state = self.list_state.clone(); @@ -15457,8 +15766,19 @@ impl Render for ThreadView { }); v_flex() - .key_context("AcpThread") + .key_context(Self::thread_key_context( + voice_key_context, + cx.platform_info().os, + )) .track_focus(&self.focus_handle) + // A disabled feature leaves the voice shortcut unbound (R1.2). + .when(voice_enabled, |this| { + this.on_action(cx.listener(Self::toggle_voice_recording)) + .on_key_up(cx.listener(Self::voice_key_up)) + .on_modifiers_changed(cx.listener(Self::voice_modifiers_changed)) + }) + .on_action(cx.listener(Self::cancel_voice_recording)) + .on_action(cx.listener(Self::dismiss_voice_download)) .on_action(cx.listener(|this, _: &menu::Cancel, _, cx| { if this.parent_session_id.is_none() { this.cancel_generation(cx); diff --git a/crates/agent_ui/src/message_editor.rs b/crates/agent_ui/src/message_editor.rs index 022ba92..b3087f8 100644 --- a/crates/agent_ui/src/message_editor.rs +++ b/crates/agent_ui/src/message_editor.rs @@ -227,6 +227,9 @@ pub struct MessageEditor { /// Shared with the editor's [`MessageEditorAddon`], which reads it into the /// key context as `question_pending`. question_pending: Rc>, + /// Shared with the editor's [`MessageEditorAddon`], which reads it into the + /// key context as `voice_recording` or `voice_consent`. + voice_key_context: Rc>, /// Set when Enter (or a send chord) was swallowed because a question is /// pending; the composer then says why until the question ends. swallowed_send: bool, @@ -512,6 +515,7 @@ impl MessageEditor { .map(|project| project.read(cx).languages().clone()); let question_pending = Rc::new(Cell::new(false)); + let voice_key_context = Rc::new(Cell::new(VoiceKeyContext::None)); let editor = cx.new(|cx| { let buffer = cx.new(|cx| { let buffer = Buffer::local("", cx); @@ -535,7 +539,10 @@ impl MessageEditor { max_entries_visible: 12, placement: None, }); - editor.register_addon(MessageEditorAddon::new(question_pending.clone())); + editor.register_addon(MessageEditorAddon::new( + question_pending.clone(), + voice_key_context.clone(), + )); editor.set_custom_context_menu(|editor, _point, window, cx| { let has_selection = editor.has_non_empty_selection(&editor.display_snapshot(cx)); @@ -659,6 +666,7 @@ impl MessageEditor { session_capabilities, local_commands, question_pending, + voice_key_context, swallowed_send: false, skip_slash_command_check: false, agent_id, @@ -682,6 +690,12 @@ impl MessageEditor { self.question_pending.set(question_pending); } + /// Which voice-input state this composer's key context names, so Escape + /// is claimed only while it has something to cancel (story 044). + pub fn set_voice_key_context(&self, voice_key_context: VoiceKeyContext) { + self.voice_key_context.set(voice_key_context); + } + /// Whether the composer is explaining a swallowed Enter: one was pressed /// while a question was pending, and the question is still pending. pub fn shows_swallowed_send_hint(&self) -> bool { @@ -2205,6 +2219,53 @@ impl MessageEditor { }); } + /// Inserts a voice transcript in place of the newest selection, as one undo step, without + /// sending it. + /// + /// The other selections are collapsed into the newest one, and a non-empty selection is + /// replaced, as typing would. A space is put before the transcript when the character before + /// the selection's start is not whitespace. A read-only composer is left untouched and the + /// refusal is returned. + pub fn insert_transcript( + &mut self, + text: &str, + window: &mut Window, + cx: &mut Context, + ) -> Result<()> { + let editor = self.editor.read(cx); + if editor.read_only(cx) { + return Err(anyhow!( + "The composer is read-only; the transcript was not inserted" + )); + } + + // Like typing, the transcript replaces a selection, so the space is decided by the + // character before the selection's start, whichever direction it was made in. + let newest = *editor.selections.newest_anchor(); + let snapshot = editor.buffer().read(cx).snapshot(cx); + let previous = snapshot.reversed_chars_at(newest.start).next(); + let insertion = voice_input::transcript::for_insertion(previous, text); + if insertion.is_empty() { + return Ok(()); + } + + self.editor.update(cx, |editor, cx| { + // Closing the transactions on both sides keeps the transcript from being grouped + // with text typed just before or after it, so one undo removes exactly it. + editor.finalize_last_transaction(cx); + editor.transact(window, cx, |editor, window, cx| { + // Only the newest cursor receives the transcript; the collapse is inside the + // transaction, so the same undo brings the other cursors back. + editor.change_selections(Default::default(), window, cx, |selections| { + selections.select_anchor_ranges([newest.start..newest.end]); + }); + editor.insert(&insertion, window, cx); + }); + editor.finalize_last_transaction(cx); + }); + Ok(()) + } + pub fn set_placeholder_text( &mut self, placeholder: &str, @@ -2447,13 +2508,32 @@ impl Render for MessageEditor { } } +/// The voice-input state a composer's key context names. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum VoiceKeyContext { + #[default] + None, + /// This composer is recording: Escape cancels the recording. + Recording, + /// This composer asks for the model download: Escape declines it. Not + /// while the download runs, so Escape keeps its usual meaning then. + Consent, +} + pub struct MessageEditorAddon { question_pending: Rc>, + voice_key_context: Rc>, } impl MessageEditorAddon { - pub fn new(question_pending: Rc>) -> Self { - Self { question_pending } + pub fn new( + question_pending: Rc>, + voice_key_context: Rc>, + ) -> Self { + Self { + question_pending, + voice_key_context, + } } } @@ -2474,6 +2554,11 @@ impl Addon for MessageEditorAddon { if self.question_pending.get() { key_context.add("question_pending"); } + match self.voice_key_context.get() { + VoiceKeyContext::None => {} + VoiceKeyContext::Recording => key_context.add("voice_recording"), + VoiceKeyContext::Consent => key_context.add("voice_consent"), + } } } diff --git a/crates/agent_ui/src/voice_button.rs b/crates/agent_ui/src/voice_button.rs new file mode 100644 index 0000000..69664da --- /dev/null +++ b/crates/agent_ui/src/voice_button.rs @@ -0,0 +1,337 @@ +//! The composer's microphone button and the strip above the editor (design.md C2). +//! +//! What each [`ViewState`] shows is decided by the pure helpers below, so the decisions can be +//! tested without inspecting a rendered element; the render functions only assemble them. + +use std::time::Duration; + +use gpui::{ + AnyElement, App, ClickEvent, EntityId, FocusHandle, KeyBinding, Keystroke, Rems, Role, + SharedString, rems, +}; +use ui::{ + Callout, CalloutBorderPosition, CommonAnimationExt as _, Icon, IconButton, ProgressBar, + Severity, TintColor, Tooltip, prelude::*, text_for_keybinding_keystrokes, +}; +use voice_input::state::{Input, Owner}; +use voice_input::{ToggleRecording, ViewState, VoiceInput, VoiceModel}; + +const BUTTON_ID: &str = "voice-input"; +const LABEL: &str = "Voice input"; +const READ_ONLY: &str = "The composer is read-only"; +const BUSY: &str = "Voice input is busy in another thread"; +const MODEL_HOST: &str = "huggingface.co"; +/// Wide enough for the icon and `m:ss` up to the 2-minute limit (R2.6). +const RECORDING_WIDTH: Rems = rems(4.5); + +/// The microphone button for the composer `owner`, or nothing while voice input is disabled +/// (R1.2). The shortcut shown in the tooltip is read from the keymap, preferring the bindings +/// active where the composer has focus. `remote_host` is [`VoiceInput::remote_host`]. +pub fn render_voice_button( + state: &ViewState, + read_only: bool, + owner: EntityId, + remote_host: Option<&str>, + focus: &FocusHandle, + window: &Window, + cx: &App, +) -> Option { + // A disabled feature costs no keymap walk on the composer's every render. + if matches!(state, ViewState::Hidden) { + return None; + } + let binding = toggle_binding(window, focus) + .map(|binding| text_for_keybinding_keystrokes(binding.keystrokes(), cx)); + let tooltip = SharedString::from(tooltip_text_with_host( + state, + read_only, + binding.as_deref(), + remote_host, + )); + let label = SharedString::from(aria_label(state)); + let disabled = is_disabled(state, read_only); + let on_click = move |_: &ClickEvent, _: &mut Window, cx: &mut App| { + send( + Input::Click { + owner: voice_owner(owner), + }, + cx, + ); + }; + + let element = match state { + ViewState::Hidden => return None, + // R2.4: tinted, with the elapsed time; it stays clickable so a click stops (R2.3). The + // fixed width keeps the ticking label from moving or wrapping the toolbar. + ViewState::Recording { elapsed } => Button::new(BUTTON_ID, elapsed_label(*elapsed)) + .start_icon( + Icon::new(IconName::Mic) + .size(IconSize::Small) + .color(Color::Error), + ) + .label_size(LabelSize::Small) + .color(Color::Error) + .style(ButtonStyle::Tinted(TintColor::Error)) + .width(RECORDING_WIDTH) + .aria_label(label) + .disabled(disabled) + .tooltip(Tooltip::text(tooltip)) + .on_click(on_click) + .into_any_element(), + // R4.5/R5.2: a spinner, which takes no click and no focus. + ViewState::Transcribing => div() + .id(BUTTON_ID) + .px_1() + .role(Role::ProgressIndicator) + .aria_label(label) + .tooltip(Tooltip::text(tooltip)) + .child( + Icon::new(IconName::LoadCircle) + .size(IconSize::Small) + .color(Color::Muted) + .with_rotate_animation(2), + ) + .into_any_element(), + ViewState::Idle { .. } + | ViewState::Busy + | ViewState::AwaitingConsent { .. } + | ViewState::Downloading { .. } => IconButton::new(BUTTON_ID, IconName::Mic) + .icon_size(IconSize::Small) + .icon_color(Color::Muted) + .aria_label(label) + .disabled(disabled) + .tooltip(Tooltip::text(tooltip)) + .on_click(on_click) + .into_any_element(), + }; + Some(element) +} + +/// The voice shortcut the tooltip shows: the binding of highest precedence, preferring the +/// bindings active where the composer has focus. +pub fn toggle_binding(window: &Window, focus: &FocusHandle) -> Option { + window + .highest_precedence_binding_for_action_in(&ToggleRecording, focus) + .or_else(|| window.highest_precedence_binding_for_action(&ToggleRecording)) +} + +/// Every binding of the voice shortcut, with the same preference and fallback as +/// [`toggle_binding`]. The key-up handler matches a release against all of them: a user's second +/// binding on another key presses the shortcut, so its key-up has to release it. +pub fn toggle_bindings(window: &Window, focus: &FocusHandle) -> Vec { + let bindings = window.bindings_for_action_in(&ToggleRecording, focus); + if bindings.is_empty() { + window.bindings_for_action(&ToggleRecording) + } else { + bindings + } +} + +/// Whether `released` lets go of the shortcut `binding`: its last keystroke's non-modifier key, +/// whatever modifiers are still down. The chord may be released in any order, and X11 sends no +/// key-up for a modifier key itself. +pub fn releases_binding(released: &Keystroke, binding: &KeyBinding) -> bool { + binding + .keystrokes() + .last() + .is_some_and(|keystroke| keystroke.inner().key.eq_ignore_ascii_case(&released.key)) +} + +/// The strip above the editor: the download consent (R6.1, R6.7) or the download's progress +/// with its Cancel control (R6.3). Nothing in every other state. +/// +/// Its controls are tab stops, so they can be reached and pressed without a mouse (M1). The +/// border sits below it: the composer already draws the rule above. +pub fn render_voice_strip(state: &ViewState, cx: &App) -> Option { + match state { + // `Callout` draws its severity's icon; `.icon(..)` only turns it on. + ViewState::AwaitingConsent { model, size } => Some( + Callout::new() + .icon(IconName::Info) + .severity(Severity::Info) + .border_position(CalloutBorderPosition::Bottom) + .title(consent_text(*model, size)) + .actions_slot( + h_flex() + .gap_1() + .child( + Button::new("voice-input-consent-decline", "Not now") + .label_size(LabelSize::Small) + .tab_index(0_isize) + .on_click(|_, _, cx| send(Input::ConsentDeclined, cx)), + ) + .child( + Button::new("voice-input-consent-download", "Download") + .label_size(LabelSize::Small) + .style(ButtonStyle::Filled) + .tab_index(0_isize) + .on_click(|_, _, cx| send(Input::ConsentGiven, cx)), + ), + ) + .into_any_element(), + ), + ViewState::Downloading { percent } => Some( + Callout::new() + .icon(IconName::Info) + .severity(Severity::Info) + .border_position(CalloutBorderPosition::Bottom) + .title("Downloading the speech model") + .actions_slot( + div() + .debug_selector(|| "voice-input-download-cancel".into()) + .child( + Button::new("voice-input-download-cancel", "Cancel") + .label_size(LabelSize::Small) + .tab_index(0_isize) + .on_click(|_, _, cx| send(Input::DownloadCancel, cx)), + ), + ) + .description_slot( + h_flex() + .w_full() + .gap_2() + .child(div().flex_1().child(ProgressBar::new( + "voice-input-download-progress", + f32::from(*percent), + 100., + cx, + ))) + .child( + Label::new(format!("{percent}%")) + .size(LabelSize::Small) + .color(Color::Muted), + ), + ) + .into_any_element(), + ), + ViewState::Hidden + | ViewState::Idle { .. } + | ViewState::Busy + | ViewState::Recording { .. } + | ViewState::Transcribing => None, + } +} + +/// The button's tooltip: the action and its shortcut (R1.1), the remote host when audio would +/// leave the machine (R8.3), or why the button is disabled (R1.3, R5.1). Names the host only +/// while idle, from the state itself; see [`tooltip_text_with_host`]. +pub fn tooltip_text(state: &ViewState, read_only: bool, binding: Option<&str>) -> String { + if read_only { + return READ_ONLY.to_string(); + } + match state { + ViewState::Hidden => LABEL.to_string(), + ViewState::Idle { remote_host } => { + with_host(with_binding(LABEL, binding), remote_host.as_deref()) + } + ViewState::Busy => BUSY.to_string(), + ViewState::AwaitingConsent { .. } => { + "Voice input is waiting for the model download to be confirmed".to_string() + } + ViewState::Downloading { .. } => "Voice input is downloading the speech model".to_string(), + ViewState::Recording { .. } => with_binding("Stop recording", binding), + ViewState::Transcribing => "Transcribing…".to_string(), + } +} + +/// [`tooltip_text`], also naming `remote_host` while recording and transcribing, when the audio +/// in flight goes to it (R8.3). +pub fn tooltip_text_with_host( + state: &ViewState, + read_only: bool, + binding: Option<&str>, + remote_host: Option<&str>, +) -> String { + let text = tooltip_text(state, read_only, binding); + if read_only { + return text; + } + match state { + ViewState::Recording { .. } | ViewState::Transcribing => with_host(text, remote_host), + ViewState::Hidden + | ViewState::Idle { .. } + | ViewState::Busy + | ViewState::AwaitingConsent { .. } + | ViewState::Downloading { .. } => text, + } +} + +/// What a screen reader announces for the button: the control, not its bare time. +pub fn aria_label(state: &ViewState) -> String { + match state { + ViewState::Recording { elapsed } => format!("Stop recording ({})", elapsed_label(*elapsed)), + ViewState::Transcribing => "Transcribing voice input".to_string(), + ViewState::Hidden + | ViewState::Idle { .. } + | ViewState::Busy + | ViewState::AwaitingConsent { .. } + | ViewState::Downloading { .. } => LABEL.to_string(), + } +} + +/// Whether the button takes no click: read-only (R1.3), busy elsewhere (R5.1), transcribing +/// (R4.5, R5.2), or while the strip is the control (consent, download). Idle and recording stay +/// clickable — a click is what stops a recording (R2.3). +pub fn is_disabled(state: &ViewState, read_only: bool) -> bool { + if read_only { + return true; + } + match state { + ViewState::Idle { .. } | ViewState::Recording { .. } => false, + ViewState::Hidden + | ViewState::Busy + | ViewState::AwaitingConsent { .. } + | ViewState::Downloading { .. } + | ViewState::Transcribing => true, + } +} + +/// The elapsed recording time as `m:ss`, truncated to the whole second (R2.4). +pub fn elapsed_label(elapsed: Duration) -> String { + let seconds = elapsed.as_secs(); + format!("{}:{:02}", seconds / 60, seconds % 60) +} + +/// The consent question: the model, its download size and the host it comes from (R6.1). +pub fn consent_text(model: VoiceModel, size: &str) -> String { + format!( + "Voice input needs the speech model {} ({size}) from {MODEL_HOST}. Download it?", + model_name(model) + ) +} + +fn model_name(model: VoiceModel) -> &'static str { + match model { + VoiceModel::Tiny => "tiny", + VoiceModel::Base => "base", + VoiceModel::Small => "small", + } +} + +fn with_binding(label: &str, binding: Option<&str>) -> String { + match binding { + Some(binding) => format!("{label} ({binding})"), + None => label.to_string(), + } +} + +fn with_host(text: String, remote_host: Option<&str>) -> String { + match remote_host { + Some(host) => format!("{text} — remote: {host}"), + None => text, + } +} + +/// The voice-input owner of the thread view `view`. +pub fn voice_owner(view: EntityId) -> Owner { + Owner(view.as_u64()) +} + +/// Hands a click or a key to the app-wide voice input. The button, the strip and the key +/// handlers are live only when it exists, so its absence here means it was torn down meanwhile. +pub fn send(input: Input, cx: &mut App) { + match VoiceInput::try_global(cx) { + Some(voice) => voice.update(cx, |voice, cx| voice.handle(input, cx)), + None => log::warn!("voice_input event=input_dropped reason=not_initialized"), + } +} diff --git a/crates/agent_ui/src/voice_button_more_tests.rs b/crates/agent_ui/src/voice_button_more_tests.rs new file mode 100644 index 0000000..0cf1871 --- /dev/null +++ b/crates/agent_ui/src/voice_button_more_tests.rs @@ -0,0 +1,122 @@ +// Story 044, task 5.2 fix round 1 — the tech review's M1, L1 and L2 +// (.draft/reports/5.2-tech-review-rust-gpui.md). +// +// Hostile halves: the strip's controls are tab stops for consent AND for download AND there is +// none when no strip shows; the remote host is named while recording AND transcribing AND not in +// local mode; the recording button is announced as a stop control AND not as its bare time. + +use std::time::Duration; + +use gpui::{Context, IntoElement, Render, TestAppContext, Window, div, prelude::*}; +use voice_input::{ViewState, VoiceModel}; + +use crate::voice_button::{aria_label, render_voice_strip, tooltip_text_with_host}; + +const HOST: &str = "api.groq.com"; + +fn recording() -> ViewState { + ViewState::Recording { + elapsed: Duration::from_secs(5), + } +} + +#[test] +fn voice_button_more_remote_host_named_while_recording_and_transcribing() { + let shown = tooltip_text_with_host(&recording(), false, Some("ctrl-alt-v"), Some(HOST)); + assert!(shown.contains("ctrl-alt-v"), "R1.1: {shown}"); + assert!( + shown.contains("remote") && shown.contains(HOST), + "R8.3 recording: {shown}" + ); + + let shown = tooltip_text_with_host(&ViewState::Transcribing, false, None, Some(HOST)); + assert!( + shown.contains("remote") && shown.contains(HOST), + "R8.3 transcribing: {shown}" + ); + + for state in [recording(), ViewState::Transcribing] { + let shown = tooltip_text_with_host(&state, false, Some("ctrl-alt-v"), None); + assert!( + !shown.is_empty() && !shown.contains("remote"), + "local {state:?}: {shown}" + ); + } +} + +#[test] +fn voice_button_more_idle_keeps_its_own_host() { + let remote = ViewState::Idle { + remote_host: Some(HOST.into()), + }; + let shown = tooltip_text_with_host(&remote, false, Some("ctrl-alt-v"), None); + assert!(shown.contains(HOST), "{shown}"); + let local = ViewState::Idle { remote_host: None }; + assert_eq!( + tooltip_text_with_host(&local, false, Some("ctrl-alt-v"), None), + "Voice input (ctrl-alt-v)" + ); + assert_eq!( + tooltip_text_with_host(&recording(), true, None, Some(HOST)), + "The composer is read-only", + "R1.3 outranks the host" + ); +} + +#[test] +fn voice_button_more_aria_labels() { + assert_eq!( + aria_label(&ViewState::Idle { remote_host: None }), + "Voice input" + ); + assert_eq!(aria_label(&recording()), "Stop recording (0:05)"); + let transcribing = aria_label(&ViewState::Transcribing); + assert!(transcribing.contains("Transcribing"), "{transcribing}"); +} + +struct StripHost(ViewState); + +impl Render for StripHost { + fn render(&mut self, _window: &mut Window, cx: &mut Context) -> impl IntoElement { + div().children(render_voice_strip(&self.0, cx)) + } +} + +/// How many distinct elements Tab cycles through in a window showing the strip for `state`. +fn tab_stops(state: ViewState, cx: &mut TestAppContext) -> usize { + let (_view, cx) = cx.add_window_view(|_, _| StripHost(state)); + cx.run_until_parked(); + let mut seen: Vec = Vec::new(); + for _ in 0..4 { + let focused = cx.update(|window, cx| { + window.focus_next(cx); + window.focused(cx) + }); + if let Some(handle) = focused + && !seen.contains(&handle) + { + seen.push(handle); + } + } + seen.len() +} + +#[gpui::test] +fn voice_button_more_strip_controls_are_tab_stops(cx: &mut TestAppContext) { + crate::test_support::init_test(cx); + let consent = ViewState::AwaitingConsent { + model: VoiceModel::Base, + size: "148 MB".into(), + }; + assert_eq!(tab_stops(consent, cx), 2, "M1: Not now and Download"); + assert_eq!( + tab_stops(ViewState::Downloading { percent: 42 }, cx), + 1, + "M1: Cancel" + ); + assert_eq!( + tab_stops(ViewState::Idle { remote_host: None }, cx), + 0, + "no strip, no stop" + ); +} diff --git a/crates/agent_ui/src/voice_button_tests.rs b/crates/agent_ui/src/voice_button_tests.rs new file mode 100644 index 0000000..5a5921d --- /dev/null +++ b/crates/agent_ui/src/voice_button_tests.rs @@ -0,0 +1,152 @@ +// Story 044, task 5.2 — what the microphone button and the strip show (R1.1, R1.3, R2.4, R4.5, R5.2, +// R6.1, R6.3, R8.3). Wired in the crate root `src/agent_ui.rs` as +// +// #[cfg(test)] +// mod voice_button_tests; +// +// Validation filter: `voice_button` (every test path is `voice_button_tests::…`). +// +// Contract: design.md C2 (`render_voice_button(view_state, focus, cx)`, +// `render_voice_strip(view_state, cx) -> Option`, the tooltips) and C1.10 `ViewState`. +// A rendered `AnyElement` cannot be inspected, so the decisions are assumed to live in pure helpers +// in `crate::voice_button` that the render functions call — adjust the `use` only: +// - `tooltip_text(state: &ViewState, read_only: bool, binding: Option<&str>) -> String` +// - `is_disabled(state: &ViewState, read_only: bool) -> bool` +// - `elapsed_label(elapsed: Duration) -> String` — `m:ss` +// - `consent_text(model: VoiceModel, size: &str) -> String` +// Presence in the real toolbar is proven by the GUI proof (7.3). +// +// Hostile halves: `m:ss` pads seconds (1:05, not 1:5) AND rolls over at 60 (1:00, not 0:60); the +// button is disabled while transcribing, busy or read-only AND enabled while idle or recording (the +// stop click must work); the tooltip names the remote host only in remote mode; the strip exists +// for consent and download AND not for idle or recording. + +use std::time::Duration; + +use gpui::TestAppContext; +use voice_input::{ViewState, models::VoiceModel}; + +use crate::voice_button::{ + consent_text, elapsed_label, is_disabled, render_voice_strip, tooltip_text, +}; + +fn idle_local() -> ViewState { + ViewState::Idle { remote_host: None } +} + +#[test] +fn voice_button_elapsed_is_m_ss() { + for (secs, want) in [ + (0, "0:00"), + (5, "0:05"), + (59, "0:59"), + (60, "1:00"), + (65, "1:05"), + (119, "1:59"), + (120, "2:00"), + ] { + assert_eq!( + elapsed_label(Duration::from_secs(secs)), + want, + "R2.4 {secs} s" + ); + } + assert_eq!( + elapsed_label(Duration::from_millis(59_999)), + "0:59", + "truncates, never shows 0:60" + ); +} + +#[test] +fn voice_button_tooltip_names_action_and_shortcut() { + assert_eq!( + tooltip_text(&idle_local(), false, Some("ctrl-alt-v")), + "Voice input (ctrl-alt-v)", + "R1.1" + ); + let without = tooltip_text(&idle_local(), false, None); + assert!(without.starts_with("Voice input"), "{without}"); +} + +#[test] +fn voice_button_tooltip_names_the_remote_host_only_when_remote() { + let remote = ViewState::Idle { + remote_host: Some("api.groq.com".into()), + }; + let shown = tooltip_text(&remote, false, Some("ctrl-alt-v")); + assert!( + shown.contains("remote") && shown.contains("api.groq.com"), + "R8.3: {shown}" + ); + assert!(!tooltip_text(&idle_local(), false, Some("ctrl-alt-v")).contains("remote")); +} + +#[test] +fn voice_button_read_only_and_busy_tooltips() { + assert_eq!( + tooltip_text(&idle_local(), true, Some("ctrl-alt-v")), + "The composer is read-only", + "R1.3" + ); + assert_eq!( + tooltip_text(&ViewState::Busy, false, None), + "Voice input is busy in another thread", + "R5.1" + ); +} + +#[test] +fn voice_button_disabled_states() { + assert!( + is_disabled(&ViewState::Transcribing, false), + "R4.5/R5.2 disabled while transcribing" + ); + assert!(is_disabled(&ViewState::Busy, false), "busy elsewhere"); + assert!(is_disabled(&idle_local(), true), "R1.3 read-only"); + assert!(!is_disabled(&idle_local(), false), "idle is clickable"); + assert!( + !is_disabled( + &ViewState::Recording { + elapsed: Duration::from_secs(3) + }, + false + ), + "R2.3 recording must stay clickable to stop" + ); +} + +#[test] +fn voice_button_consent_names_model_size_and_host() { + let text = consent_text(VoiceModel::Base, "148 MB"); + for needle in ["base", "148 MB", "huggingface.co"] { + assert!(text.contains(needle), "R6.1 {needle}: {text}"); + } +} + +#[gpui::test] +fn voice_button_strip_only_for_consent_and_download(cx: &mut TestAppContext) { + crate::test_support::init_test(cx); + let cx = cx.add_empty_window(); + let consent = ViewState::AwaitingConsent { + model: VoiceModel::Base, + size: "148 MB".into(), + }; + let downloading = ViewState::Downloading { percent: 42 }; + for (state, present) in [ + (consent, true), + (downloading, true), + (idle_local(), false), + ( + ViewState::Recording { + elapsed: Duration::from_secs(1), + }, + false, + ), + (ViewState::Transcribing, false), + (ViewState::Hidden, false), + ] { + let shown = cx.update(|_window, cx| render_voice_strip(&state, cx).is_some()); + assert_eq!(shown, present, "R6.1/R6.3 strip for {state:?}"); + } +} diff --git a/crates/agent_ui/src/voice_contract_tests.rs b/crates/agent_ui/src/voice_contract_tests.rs new file mode 100644 index 0000000..49aa7e7 --- /dev/null +++ b/crates/agent_ui/src/voice_contract_tests.rs @@ -0,0 +1,110 @@ +// Story 044, task 6.1 — the contract between `voice_input` and the four keymaps. Every +// `voice_input::` action a keymap names must resolve, so a misspelled name cannot sit unbound, +// and `DismissDownload` (added by 5.3) is only ever claimed while the consent is asked. +// Validation filter: `voice` (test paths are `voice_contract_tests::…`). + +use gpui::TestAppContext; + +use crate::test_support::init_test; + +const KEYMAPS: [(&str, &str); 4] = [ + ( + "keymaps/default-linux.json", + include_str!("../../../assets/keymaps/default-linux.json"), + ), + ( + "keymaps/default-windows.json", + include_str!("../../../assets/keymaps/default-windows.json"), + ), + ( + "keymaps/default-macos.json", + include_str!("../../../assets/keymaps/default-macos.json"), + ), + ( + "keymaps/vim.json", + include_str!("../../../assets/keymaps/vim.json"), + ), +]; + +const ACTIONS: [&str; 3] = [ + "voice_input::ToggleRecording", + "voice_input::CancelRecording", + "voice_input::DismissDownload", +]; + +/// Every `(context, keystroke, action)` whose action is in the `voice_input` namespace. +fn voice_bindings(content: &str) -> Vec<(String, String, String)> { + let sections = + settings::parse_json_with_comments::>(content).expect("keymap JSON"); + let mut found = Vec::new(); + for section in sections { + let context = section + .get("context") + .and_then(|c| c.as_str()) + .unwrap_or("") + .to_string(); + let Some(bindings) = section.get("bindings").and_then(|b| b.as_object()) else { + continue; + }; + for (keys, value) in bindings { + if let Some(action) = value.as_str().filter(|a| a.starts_with("voice_input::")) { + found.push((context.clone(), keys.clone(), action.to_string())); + } + } + } + found +} + +#[test] +fn voice_contract_keymaps_name_only_known_voice_actions() { + for (name, content) in KEYMAPS { + for (context, keys, action) in voice_bindings(content) { + assert!( + ACTIONS.contains(&action.as_str()), + "{name}: {keys:?} in {context:?} names an unknown voice action {action}" + ); + } + } +} + +#[test] +fn voice_contract_dismiss_download_is_escape_under_voice_consent_only() { + for (name, content) in KEYMAPS { + let dismiss: Vec<_> = voice_bindings(content) + .into_iter() + .filter(|(_, _, action)| action == "voice_input::DismissDownload") + .collect(); + assert!( + dismiss + .iter() + .any(|(context, keys, _)| keys == "escape" && context.contains("Editor")), + "{name}: escape → DismissDownload in the composer, found {dismiss:?}" + ); + assert!( + dismiss + .iter() + .all(|(context, keys, _)| keys == "escape" && context.contains("voice_consent")), + "{name}: DismissDownload only on escape under voice_consent, found {dismiss:?}" + ); + } +} + +#[gpui::test] +fn voice_contract_every_voice_action_resolves_in_every_keymap(cx: &mut TestAppContext) { + init_test(cx); + for (name, _) in KEYMAPS { + let names: Vec = cx.update(|cx| { + settings::KeymapFile::load_asset_allow_partial_failure(name, cx) + .expect("loads") + .iter() + .map(|binding| binding.action().name().to_string()) + .collect() + }); + for action in ACTIONS { + assert!( + names.iter().any(|n| n == action), + "{name}: {action} resolves to an action" + ); + } + } +} diff --git a/crates/agent_ui/src/voice_download_adopt_ui_tests.rs b/crates/agent_ui/src/voice_download_adopt_ui_tests.rs new file mode 100644 index 0000000..494af2c --- /dev/null +++ b/crates/agent_ui/src/voice_download_adopt_ui_tests.rs @@ -0,0 +1,232 @@ +// Story 044, task 8.5 fix round 1 — the tech review's I1 (.draft/reports/8.5-tech-review-rust-gpui.md): +// R6.15 must be reachable from the real composer, not only through an input sent to the entity. +// Wired in the crate root `src/agent_ui.rs` beside the other voice suites. Validation filter: +// `voice_download_adopt_ui` (every test path is `voice_download_adopt_ui_tests::…`). +// +// A disabled button delivers no click (`button_like.rs` filters `on_click` on `disabled`), so a +// composer showing `Busy` for a download whose thread is gone could never adopt it with the mouse. +// Once the confirming composer is closed, every other composer shows the download's progress and +// its Cancel, and that Cancel cancels it. +// +// Hostile halves: while the confirming thread is open, another composer is still busy (no strip); +// once it is closed, the other composer draws the progress strip AND its Cancel, clicked with the +// mouse, ends the download (the body reader is dropped, one request, idle again). + +use std::{ + collections::VecDeque, + pin::Pin, + sync::{ + Arc, Mutex, + atomic::{AtomicUsize, Ordering}, + }, + task::{Context, Poll, Waker}, + time::{Duration, Instant}, +}; + +use futures::AsyncRead; +use gpui::{AppContext as _, Modifiers, TestAppContext, VisualTestContext}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use voice_input::state::Input; +use voice_input::{ViewState, VoiceInput}; + +use crate::voice_button::{send, voice_owner}; +use crate::voice_keys_tests::{Thread, init_voice, open_thread}; + +/// Local mode with no model on disk: the first press asks for the download (R6.1). +const LOCAL_SETTINGS: &str = r#"{ "voice_input": { "model": "tiny" } }"#; +const CANCEL: &str = "voice-input-download-cancel"; +/// Longer than the download's 250 ms progress interval, so the next bytes are reported. +const PROGRESS_INTERVAL: Duration = Duration::from_millis(300); + +// ── a download body fed by the test ───────────────────────────────────────── + +#[derive(Default)] +struct Feed { + queue: VecDeque, + waker: Option, + read: u64, + dropped: bool, +} + +#[derive(Clone, Default)] +struct Body(Arc>); + +impl Body { + fn push(&self, bytes: usize) { + let mut feed = self.0.lock().expect("lock"); + feed.queue.extend(std::iter::repeat_n(0x5a, bytes)); + if let Some(waker) = feed.waker.take() { + waker.wake(); + } + } + fn read(&self) -> u64 { + self.0.lock().expect("lock").read + } + fn dropped(&self) -> bool { + self.0.lock().expect("lock").dropped + } +} + +struct Reader(Body); + +impl AsyncRead for Reader { + fn poll_read( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut [u8], + ) -> Poll> { + let mut feed = (self.0).0.lock().expect("lock"); + if feed.queue.is_empty() { + feed.waker = Some(cx.waker().clone()); + return Poll::Pending; + } + let count = buf.len().min(feed.queue.len()); + for (slot, byte) in buf.iter_mut().zip(feed.queue.drain(..count)) { + *slot = byte; + } + feed.read += count as u64; + Poll::Ready(Ok(count)) + } +} + +impl Drop for Reader { + fn drop(&mut self) { + if let Ok(mut feed) = (self.0).0.lock() { + feed.dropped = true; + } + } +} + +fn fed_download(body: &Body, requests: &Arc) -> Arc { + let body = body.clone(); + let requests = requests.clone(); + FakeHttpClient::create(move |_| { + requests.fetch_add(1, Ordering::SeqCst); + let reader = Reader(body.clone()); + async move { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from_reader(reader))?) + } + }) +} + +// ── helpers ───────────────────────────────────────────────────────────────── + +fn view_state(thread: &Thread, cx: &TestAppContext) -> ViewState { + let owner = thread.view(cx).entity_id(); + cx.read(|cx| VoiceInput::global(cx).read(cx).view_state(owner)) +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +/// Feeds `bytes` and waits until the download has read them; each read is a progress update, +/// which re-renders every composer. +fn feed(body: &Body, bytes: usize, cx: &mut TestAppContext) { + let target = body.read() + bytes as u64; + body.push(bytes); + wait_until(cx, "the download to read the fed bytes", |_| { + body.read() >= target + }); +} + +// ── I1 ────────────────────────────────────────────────────────────────────── + +#[gpui::test] +async fn voice_download_adopt_ui_a_closed_threads_download_shows_its_cancel_in_another_composer( + cx: &mut TestAppContext, +) { + let body = Body::default(); + let requests = Arc::new(AtomicUsize::new(0)); + let _voice = init_voice(LOCAL_SETTINGS, fed_download(&body, &requests), cx); + + // Thread A asks for the model and the user confirms there. + let a = open_thread(cx).await; + let owner_a = voice_owner(a.view(cx).entity_id()); + cx.update(|cx| send(Input::Click { owner: owner_a }, cx)); + cx.run_until_parked(); + assert!( + matches!(view_state(&a, cx), ViewState::AwaitingConsent { .. }), + "precondition: A is asked for consent: {:?}", + view_state(&a, cx) + ); + cx.update(|cx| send(Input::ConsentGiven, cx)); + wait_until(cx, "the download request", |_| { + requests.load(Ordering::SeqCst) == 1 + }); + feed(&body, 64 * 1024, cx); + assert!( + matches!(view_state(&a, cx), ViewState::Downloading { .. }), + "precondition: A's download runs: {:?}", + view_state(&a, cx) + ); + + // While A is open, B is busy: the download is A's. + let b = open_thread(cx).await; + let mut vcx_b: VisualTestContext = b.vcx(cx); + vcx_b.run_until_parked(); + assert!( + matches!(view_state(&b, cx), ViewState::Busy), + "R5.1 B is busy while A, open, downloads: {:?}", + view_state(&b, cx) + ); + assert!( + vcx_b.debug_bounds(CANCEL).is_none(), + "no strip in B while A owns the download" + ); + + // A is closed: its window removed and every strong handle the test holds dropped. + let a_view = a.view(cx).downgrade(); + cx.update_window(a.window, |_, window, _| window.remove_window()) + .expect("close A"); + drop(a); + // The released window's entities go at the next effect flush. + cx.update(|_| {}); + cx.run_until_parked(); + assert!( + a_view.upgrade().is_none(), + "precondition: A's thread view is released" + ); + // Progress is reported at most every 250 ms; the next report re-renders every composer. + std::thread::sleep(PROGRESS_INTERVAL); + feed(&body, 64 * 1024, cx); + + assert!( + matches!(view_state(&b, cx), ViewState::Downloading { .. }), + "R6.15 B shows the kept download, not busy for a thread that is gone: {:?}", + view_state(&b, cx) + ); + vcx_b.run_until_parked(); + let cancel = vcx_b + .debug_bounds(CANCEL) + .expect("R6.3 B's strip draws the download's Cancel"); + assert!(!body.dropped(), "R6.14 the download still runs"); + + vcx_b.simulate_click(cancel.center(), Modifiers::default()); + vcx_b.run_until_parked(); + wait_until(cx, "the download to stop", |_| body.dropped()); + assert!( + matches!(view_state(&b, cx), ViewState::Idle { .. }), + "R6.6 idle after B's Cancel: {:?}", + view_state(&b, cx) + ); + assert_eq!( + requests.load(Ordering::SeqCst), + 1, + "one request, never restarted" + ); +} diff --git a/crates/agent_ui/src/voice_events_more_tests.rs b/crates/agent_ui/src/voice_events_more_tests.rs new file mode 100644 index 0000000..7b6b431 --- /dev/null +++ b/crates/agent_ui/src/voice_events_more_tests.rs @@ -0,0 +1,59 @@ +// Story 044, task 5.4 fix round 1 (tech review L3) — `VoiceEvent::Changed` re-renders the thread +// view, and an event meant for another owner does not. Wired in the crate root `src/agent_ui.rs`. +// Validation filter: `voice_events` (every test path is `voice_events_more_tests::…`). +// +// A view re-renders when it is notified, so the test counts the thread view's notifications. + +use std::sync::{ + Arc, + atomic::{AtomicUsize, Ordering}, +}; + +use gpui::{AppContext as _, TestAppContext}; +use voice_input::{VoiceEvent, VoiceInput}; + +use crate::voice_keys_tests::{REMOTE_SETTINGS, answering, init_voice, open_thread}; + +fn emit(event: VoiceEvent, cx: &mut TestAppContext) { + let voice = cx.update(|cx| VoiceInput::global(cx)); + voice.update(cx, |_, cx| cx.emit(event)); + cx.run_until_parked(); +} + +#[gpui::test] +async fn voice_events_changed_rerenders_the_thread_view(cx: &mut TestAppContext) { + let _voice = init_voice(REMOTE_SETTINGS, answering("{}"), cx); + let thread = open_thread(cx).await; + let view = thread.view(cx); + let notified = Arc::new(AtomicUsize::new(0)); + let _observation = cx.update({ + let notified = notified.clone(); + |cx| { + cx.observe(&view, move |_, _| { + notified.fetch_add(1, Ordering::SeqCst); + }) + } + }); + cx.run_until_parked(); + + let stranger = cx.update(|cx| cx.new(|_| ())); + let before = notified.load(Ordering::SeqCst); + emit( + VoiceEvent::Notify { + owner: stranger.entity_id(), + message: "nobody's".into(), + }, + cx, + ); + assert_eq!( + notified.load(Ordering::SeqCst), + before, + "another owner's event does not re-render this view" + ); + + emit(VoiceEvent::Changed, cx); + assert!( + notified.load(Ordering::SeqCst) > before, + "Changed re-renders the thread view" + ); +} diff --git a/crates/agent_ui/src/voice_events_tests.rs b/crates/agent_ui/src/voice_events_tests.rs new file mode 100644 index 0000000..5b158c1 --- /dev/null +++ b/crates/agent_ui/src/voice_events_tests.rs @@ -0,0 +1,223 @@ +// Story 044, task 5.4 — the thread view acts only on `VoiceInput` events for its own owner (R4.1, +// R4.7, R5.1; design.md C1.10 `VoiceEvent`, C2). Wired in the crate root `src/agent_ui.rs` as +// +// #[cfg(test)] +// mod voice_events_tests; +// +// Validation filter: `voice_events` (every test path is `voice_events_tests::…`). +// +// Harness: `crate::voice_keys_tests::{init_voice, open_thread, …}` (same assumed seams). The owner of +// a thread view is its `ThreadView` entity id (design C2: `owner == cx.entity_id()`), carried as +// `voice_input::state::Owner(id.as_u64())` in inputs and as the `EntityId` in `VoiceEvent`. +// Events are injected with `cx.emit` on the global entity for the routing tests; the closed-composer +// test (R4.7) runs a real recording whose remote answer is held until the composer is gone. +// +// Hostile halves: an Insert for A lands in A AND not in B, and the converse; a Notify for A shows in +// A's workspace AND an event for an owner nobody holds changes no composer; a transcript arriving +// after its composer closed is discarded with a notification AND does not land in the other one. + +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use futures::channel::oneshot; +use gpui::{AppContext as _, EntityId, TestAppContext}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use voice_input::state::{Input, Owner}; +use voice_input::{VoiceEvent, VoiceInput}; + +use crate::voice_keys_tests::{REMOTE_SETTINGS, Thread, answering, init_voice, open_thread}; + +fn owner_of(thread: &Thread, cx: &TestAppContext) -> EntityId { + thread.view(cx).entity_id() +} + +fn text_of(thread: &Thread, cx: &mut TestAppContext) -> String { + let editor = thread.editor(cx); + cx.read(|cx| editor.read(cx).text(cx)) +} + +fn notifications(thread: &Thread, cx: &mut TestAppContext) -> usize { + thread + .workspace + .read_with(cx, |w, _| w.notification_ids().len()) +} + +fn emit(event: VoiceEvent, cx: &mut TestAppContext) { + let vi = cx.update(|cx| VoiceInput::global(cx)); + vi.update(cx, |_, cx| cx.emit(event)); + cx.run_until_parked(); +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +#[gpui::test] +async fn voice_events_insert_lands_only_in_its_owner(cx: &mut TestAppContext) { + let _voice = init_voice(REMOTE_SETTINGS, answering("{}"), cx); + let a = open_thread(cx).await; + let b = open_thread(cx).await; + let (ida, idb) = (owner_of(&a, cx), owner_of(&b, cx)); + + emit( + VoiceEvent::Insert { + owner: ida, + text: "for A".into(), + }, + cx, + ); + assert_eq!(text_of(&a, cx), "for A", "R4.1"); + assert_eq!(text_of(&b, cx), "", "B ignores A's transcript"); + + emit( + VoiceEvent::Insert { + owner: idb, + text: "for B".into(), + }, + cx, + ); + assert_eq!(text_of(&b, cx), "for B"); + assert_eq!(text_of(&a, cx), "for A", "A ignores B's transcript"); +} + +#[gpui::test] +async fn voice_events_unknown_owner_changes_nothing(cx: &mut TestAppContext) { + let _voice = init_voice(REMOTE_SETTINGS, answering("{}"), cx); + let a = open_thread(cx).await; + let stranger = cx.update(|cx| { + use gpui::AppContext as _; + cx.new(|_| ()) + }); + let before = notifications(&a, cx); + emit( + VoiceEvent::Insert { + owner: stranger.entity_id(), + text: "nobody's".into(), + }, + cx, + ); + emit( + VoiceEvent::Notify { + owner: stranger.entity_id(), + message: "nobody's".into(), + }, + cx, + ); + assert_eq!(text_of(&a, cx), ""); + assert_eq!( + notifications(&a, cx), + before, + "A's workspace shows nothing for another owner" + ); +} + +#[gpui::test] +async fn voice_events_notify_shows_in_the_owners_workspace(cx: &mut TestAppContext) { + let _voice = init_voice(REMOTE_SETTINGS, answering("{}"), cx); + let a = open_thread(cx).await; + let b = open_thread(cx).await; + let (before_a, before_b) = (notifications(&a, cx), notifications(&b, cx)); + emit( + VoiceEvent::Notify { + owner: owner_of(&a, cx), + message: "Voice input is busy in another thread".into(), + }, + cx, + ); + assert_eq!( + notifications(&a, cx), + before_a + 1, + "R5.1 the owner is told" + ); + assert_eq!(notifications(&b, cx), before_b, "not the other composer"); +} + +#[gpui::test] +async fn voice_events_insert_into_a_read_only_composer_is_refused_visibly(cx: &mut TestAppContext) { + let _voice = init_voice(REMOTE_SETTINGS, answering("{}"), cx); + let a = open_thread(cx).await; + let editor = a.editor(cx); + editor.update(cx, |me, cx| me.set_read_only(true, cx)); + let before = notifications(&a, cx); + emit( + VoiceEvent::Insert { + owner: owner_of(&a, cx), + text: "lost".into(), + }, + cx, + ); + assert_eq!( + text_of(&a, cx), + "", + "nothing inserted into a read-only composer" + ); + assert_eq!( + notifications(&a, cx), + before + 1, + "the refusal is shown, not swallowed" + ); +} + +#[gpui::test] +async fn voice_events_transcript_for_a_closed_composer_is_discarded(cx: &mut TestAppContext) { + let gate: Arc>>> = Arc::default(); + let (release, held) = oneshot::channel(); + *gate.lock().expect("lock") = Some(held); + let g = gate.clone(); + let http: Arc = FakeHttpClient::create(move |_req| { + let held = g.lock().expect("lock").take(); + async move { + if let Some(held) = held { + let _ = held.await; + } + Ok(Response::builder() + .status(200) + .body(AsyncBody::from(r#"{"text": "orphan transcript"}"#))?) + } + }); + let voice = init_voice(REMOTE_SETTINGS, http, cx); + let a = open_thread(cx).await; + let b = open_thread(cx).await; + let ida = Owner(owner_of(&a, cx).as_u64()); + let vi = cx.update(|cx| VoiceInput::global(cx)); + + vi.update(cx, |vi, cx| vi.handle(Input::Click { owner: ida }, cx)); + wait_until(cx, "recording", |_| { + voice.mic.starts.load(std::sync::atomic::Ordering::SeqCst) == 1 + }); + vi.update(cx, |vi, cx| vi.handle(Input::Click { owner: ida }, cx)); + wait_until(cx, "the stop", |_| { + voice.mic.stops.load(std::sync::atomic::Ordering::SeqCst) == 1 + }); + + // Close A: remove its window and drop every strong handle the test holds, so the thread view + // (and its subscription) is really gone. + let workspace_a = a.workspace.clone(); + let count = move |b: &Thread, cx: &mut TestAppContext| { + workspace_a.read_with(cx, |w, _| w.notification_ids().len()) + notifications(b, cx) + }; + let before = count(&b, cx); + cx.update_window(a.window, |_, window, _| window.remove_window()) + .expect("close A"); + drop(a); + cx.run_until_parked(); + let _ = release.send(()); + wait_until(cx, "the discard notice", |cx| count(&b, cx) > before); + assert_eq!( + text_of(&b, cx), + "", + "R4.7 never delivered to another composer" + ); +} diff --git a/crates/agent_ui/src/voice_insert_more_tests.rs b/crates/agent_ui/src/voice_insert_more_tests.rs new file mode 100644 index 0000000..0024066 --- /dev/null +++ b/crates/agent_ui/src/voice_insert_more_tests.rs @@ -0,0 +1,184 @@ +// Story 044, task 5.1, fix round 1 — `MessageEditor::insert_transcript` with a selection, with +// several cursors, and next to typed text (R4.1, R4.2, R4.4; tech review of 5.1). +// +// Decision (orchestrator): dictation replaces a non-empty selection exactly like typing, and one +// undo restores the selected text. With several cursors only the newest receives the transcript. +// +// Validation filter: `voice_insert` (every test path is `voice_insert_more_tests::…`). +// +// The text buffer does not group transactions under `cfg(test)` (its group interval is zero), so +// the back-to-back tests opt in to production-like grouping; without that they would pass even if +// `insert_transcript` merged into the neighbouring undo step. + +use std::path::Path; +use std::time::Duration; + +use editor::{EditorMode, MultiBufferOffset}; +use fs::FakeFs; +use gpui::{AppContext as _, Entity, TestAppContext, VisualTestContext}; +use project::Project; +use serde_json::json; +use util::path; +use workspace::MultiWorkspace; + +use crate::message_editor::MessageEditor; +use crate::test_support::init_test; + +async fn setup(cx: &mut TestAppContext) -> (Entity, &mut VisualTestContext) { + let fs = FakeFs::new(cx.executor()); + fs.insert_tree("/project", json!({ "file.txt": "" })).await; + let project = Project::test(fs, [Path::new(path!("/project"))], cx).await; + let (multi_workspace, cx) = + cx.add_window_view(|window, cx| MultiWorkspace::test_new(project.clone(), window, cx)); + let workspace = multi_workspace.read_with(cx, |mw, _| mw.workspace().clone()); + let editor = cx.update(|window, cx| { + cx.new(|cx| { + MessageEditor::new( + workspace.downgrade(), + project.downgrade(), + None, + Default::default(), + "Test Agent".into(), + "Test", + EditorMode::AutoHeight { + min_lines: 1, + max_lines: None, + }, + window, + cx, + ) + }) + }); + cx.run_until_parked(); + (editor, cx) +} + +/// Sets the text as its own undo step and selects `ranges` (bytes); the last range is the newest. +fn prepare( + editor: &Entity, + text: &str, + ranges: &[std::ops::Range], + cx: &mut VisualTestContext, +) { + editor.update_in(cx, |me, window, cx| { + me.set_text(text, window, cx); + me.editor().update(cx, |e, cx| { + e.buffer() + .update(cx, |b, cx| b.finalize_last_transaction(cx)); + e.change_selections(Default::default(), window, cx, |s| { + s.select_ranges( + ranges + .iter() + .map(|r| MultiBufferOffset(r.start)..MultiBufferOffset(r.end)), + ) + }); + }); + }); +} + +/// Groups edits made within a minute of each other into one undo step, as the 300 ms production +/// interval does for edits made back to back. +fn group_like_production(editor: &Entity, cx: &mut VisualTestContext) { + editor.update(cx, |me, cx| { + me.editor().update(cx, |e, cx| { + e.buffer().update(cx, |b, cx| { + b.set_group_interval(Duration::from_secs(60), cx) + }) + }); + }); +} + +/// Types `typed` at the cursor the way the editor does, with no finalize around it. +fn type_text(editor: &Entity, typed: &str, cx: &mut VisualTestContext) { + editor.update_in(cx, |me, window, cx| { + me.editor().update(cx, |e, cx| e.insert(typed, window, cx)) + }); +} + +fn insert(editor: &Entity, transcript: &str, cx: &mut VisualTestContext) { + editor + .update_in(cx, |me, window, cx| { + me.insert_transcript(transcript, window, cx) + }) + .expect("inserted"); + cx.run_until_parked(); +} + +fn undo(editor: &Entity, cx: &mut VisualTestContext) { + editor.update_in(cx, |me, window, cx| { + me.editor() + .update(cx, |e, cx| e.undo(&editor::actions::Undo, window, cx)) + }); +} + +fn text(editor: &Entity, cx: &mut VisualTestContext) -> String { + editor.update(cx, |me, cx| me.text(cx)) +} + +#[gpui::test] +async fn voice_insert_replaces_a_forward_selection_with_one_space(cx: &mut TestAppContext) { + init_test(cx); + let (editor, cx) = setup(cx).await; + // "world" selected left to right: the head is after the `d` the insert removes. + prepare(&editor, "Hello world", &[6..11], cx); + insert(&editor, "there", cx); + assert_eq!( + text(&editor, cx), + "Hello there", + "R4.2 one space, selection replaced" + ); + undo(&editor, cx); + assert_eq!( + text(&editor, cx), + "Hello world", + "R4.4 one undo restores the selection" + ); +} + +#[gpui::test] +async fn voice_insert_with_two_cursors_lands_once_at_the_newest(cx: &mut TestAppContext) { + init_test(cx); + let (editor, cx) = setup(cx).await; + // The cursor at 5 is the newest although it comes first in the text. + prepare(&editor, "Hello world", &[11..11, 5..5], cx); + insert(&editor, "there", cx); + assert_eq!( + text(&editor, cx), + "Hello there world", + "R4.1 once, at the newest cursor" + ); + undo(&editor, cx); + assert_eq!(text(&editor, cx), "Hello world", "R4.4 one undo"); +} + +#[gpui::test] +async fn voice_insert_right_after_typing_undoes_only_the_transcript(cx: &mut TestAppContext) { + init_test(cx); + let (editor, cx) = setup(cx).await; + group_like_production(&editor, cx); + type_text(&editor, "Typed by hand", cx); + insert(&editor, "then dictated", cx); + assert_eq!(text(&editor, cx), "Typed by hand then dictated"); + undo(&editor, cx); + assert_eq!( + text(&editor, cx), + "Typed by hand", + "R4.4 the typed text survives" + ); +} + +#[gpui::test] +async fn voice_insert_then_typing_undoes_only_the_typing(cx: &mut TestAppContext) { + init_test(cx); + let (editor, cx) = setup(cx).await; + group_like_production(&editor, cx); + insert(&editor, "Dictated words", cx); + type_text(&editor, " and typed", cx); + assert_eq!(text(&editor, cx), "Dictated words and typed"); + undo(&editor, cx); + assert_eq!( + text(&editor, cx), + "Dictated words", + "R4.4 the transcript is its own step" + ); +} diff --git a/crates/agent_ui/src/voice_insert_tests.rs b/crates/agent_ui/src/voice_insert_tests.rs new file mode 100644 index 0000000..12319c8 --- /dev/null +++ b/crates/agent_ui/src/voice_insert_tests.rs @@ -0,0 +1,179 @@ +// Story 044, task 5.1 — `MessageEditor::insert_transcript` (R4.1, R4.2, R4.4; design.md C2 and the +// last Error Handling row). Wired in the crate root `src/agent_ui.rs` as +// +// #[cfg(test)] +// mod voice_insert_tests; +// +// Validation filter: `voice_insert` (every test path is `voice_insert_tests::…`). +// +// Contract: `insert_transcript(&mut self, text: &str, window, cx) -> anyhow::Result<()>` — refuses a +// read-only composer, puts one space before the transcript after a non-whitespace character, is one +// undo step, never sends. The setup mirrors `message_editor::tests::setup_message_editor` (private +// there); on the series tip, copy its current `MessageEditor::new` arguments if they moved. +// +// Hostile halves: one undo removes the whole transcript (space included) AND only the transcript — +// what was typed before survives; a space after a letter AND none after a space, a newline or at the +// start; insertion happens at the cursor, not at the end; a read-only composer is untouched AND the +// refusal says why; no Send/SendImmediately event, ever. + +use std::path::Path; +use std::sync::{Arc, Mutex}; + +use editor::{EditorMode, MultiBufferOffset}; +use fs::FakeFs; +use gpui::{AppContext as _, Entity, TestAppContext, VisualTestContext}; +use project::Project; +use serde_json::json; +use util::path; +use workspace::MultiWorkspace; + +use crate::message_editor::{MessageEditor, MessageEditorEvent}; +use crate::test_support::init_test; + +async fn setup( + cx: &mut TestAppContext, +) -> ( + Entity, + Arc>>, + &mut VisualTestContext, +) { + let fs = FakeFs::new(cx.executor()); + fs.insert_tree("/project", json!({ "file.txt": "" })).await; + let project = Project::test(fs, [Path::new(path!("/project"))], cx).await; + let (multi_workspace, cx) = + cx.add_window_view(|window, cx| MultiWorkspace::test_new(project.clone(), window, cx)); + let workspace = multi_workspace.read_with(cx, |mw, _| mw.workspace().clone()); + let editor = cx.update(|window, cx| { + cx.new(|cx| { + MessageEditor::new( + workspace.downgrade(), + project.downgrade(), + None, + Default::default(), + "Test Agent".into(), + "Test", + EditorMode::AutoHeight { + min_lines: 1, + max_lines: None, + }, + window, + cx, + ) + }) + }); + let sent: Arc>> = Arc::default(); + let sink = sent.clone(); + cx.update(|_, cx| { + cx.subscribe(&editor, move |_, event: &MessageEditorEvent, _| { + if matches!( + event, + MessageEditorEvent::Send | MessageEditorEvent::SendImmediately + ) { + sink.lock().expect("lock").push(format!("{event:?}")); + } + }) + .detach() + }); + cx.run_until_parked(); + (editor, sent, cx) +} + +/// Sets the text as its own undo step and puts the cursor at `cursor` (bytes). +fn prepare(editor: &Entity, text: &str, cursor: usize, cx: &mut VisualTestContext) { + editor.update_in(cx, |me, window, cx| { + me.set_text(text, window, cx); + me.editor().update(cx, |e, cx| { + e.buffer() + .update(cx, |b, cx| b.finalize_last_transaction(cx)); + e.change_selections(Default::default(), window, cx, |s| { + s.select_ranges([MultiBufferOffset(cursor)..MultiBufferOffset(cursor)]) + }); + }); + }); +} + +fn insert( + editor: &Entity, + transcript: &str, + cx: &mut VisualTestContext, +) -> anyhow::Result<()> { + let result = editor.update_in(cx, |me, window, cx| { + me.insert_transcript(transcript, window, cx) + }); + cx.run_until_parked(); + result +} + +fn text(editor: &Entity, cx: &mut VisualTestContext) -> String { + editor.update(cx, |me, cx| me.text(cx)) +} + +#[gpui::test] +async fn voice_insert_into_an_empty_composer_has_no_leading_space(cx: &mut TestAppContext) { + init_test(cx); + let (editor, sent, cx) = setup(cx).await; + insert(&editor, "Refactor the parser", cx).expect("inserted"); + assert_eq!(text(&editor, cx), "Refactor the parser"); + assert!(sent.lock().expect("lock").is_empty(), "R4.1 nothing sent"); +} + +#[gpui::test] +async fn voice_insert_after_a_letter_adds_one_space(cx: &mut TestAppContext) { + init_test(cx); + let (editor, sent, cx) = setup(cx).await; + prepare(&editor, "Fix the bug", 11, cx); + insert(&editor, "in the lexer", cx).expect("inserted"); + assert_eq!(text(&editor, cx), "Fix the bug in the lexer", "R4.2"); + assert!(sent.lock().expect("lock").is_empty()); +} + +#[gpui::test] +async fn voice_insert_after_whitespace_adds_no_space(cx: &mut TestAppContext) { + init_test(cx); + let (editor, _sent, cx) = setup(cx).await; + prepare(&editor, "Fix: ", 5, cx); + insert(&editor, "the lexer", cx).expect("inserted"); + assert_eq!(text(&editor, cx), "Fix: the lexer"); + prepare(&editor, "line one\n", 9, cx); + insert(&editor, "line two", cx).expect("inserted"); + assert_eq!(text(&editor, cx), "line one\nline two"); +} + +#[gpui::test] +async fn voice_insert_goes_to_the_cursor_not_the_end(cx: &mut TestAppContext) { + init_test(cx); + let (editor, _sent, cx) = setup(cx).await; + prepare(&editor, "Hello world", 5, cx); + insert(&editor, "there", cx).expect("inserted"); + assert_eq!(text(&editor, cx), "Hello there world", "R4.1 at the cursor"); +} + +#[gpui::test] +async fn voice_insert_is_one_undo_step_and_only_that(cx: &mut TestAppContext) { + init_test(cx); + let (editor, _sent, cx) = setup(cx).await; + prepare(&editor, "Typed by hand", 13, cx); + insert(&editor, "and then dictated words", cx).expect("inserted"); + assert_eq!(text(&editor, cx), "Typed by hand and then dictated words"); + editor.update_in(cx, |me, window, cx| { + me.editor() + .update(cx, |e, cx| e.undo(&editor::actions::Undo, window, cx)) + }); + assert_eq!( + text(&editor, cx), + "Typed by hand", + "R4.4 one undo removes the transcript and its space" + ); +} + +#[gpui::test] +async fn voice_insert_refuses_a_read_only_composer(cx: &mut TestAppContext) { + init_test(cx); + let (editor, sent, cx) = setup(cx).await; + prepare(&editor, "frozen", 6, cx); + editor.update(cx, |me, cx| me.set_read_only(true, cx)); + let err = insert(&editor, "should not land", cx).expect_err("read-only refuses"); + assert!(format!("{err:#}").contains("read-only"), "{err:#}"); + assert_eq!(text(&editor, cx), "frozen", "untouched"); + assert!(sent.lock().expect("lock").is_empty()); +} diff --git a/crates/agent_ui/src/voice_keyless_tests.rs b/crates/agent_ui/src/voice_keyless_tests.rs new file mode 100644 index 0000000..eb2de11 --- /dev/null +++ b/crates/agent_ui/src/voice_keyless_tests.rs @@ -0,0 +1,183 @@ +// Story 044, task 8.3 (refine v3) — a toggle run from the command palette is a tap (R2.3, R2.7, +// R3.2; design.md C2 "v3, keyless dispatch"). Wired in the crate root `src/agent_ui.rs` beside +// `voice_keys_tests`, whose harness it reuses: +// +// #[cfg(test)] +// mod voice_keyless_tests; +// +// Validation filter: `voice_keyless` (every test path is `voice_keyless_tests::…`). +// +// Contract: `ToggleRecording` dispatched with no keystroke (the command palette's way in) acts as +// a click: from idle it starts a toggle recording, on a running toggle it stops and transcribes, +// and nothing is left waiting for a key release. The keyed paths are unchanged: a tap toggles, a +// press held past 300 ms is a hold, a key repeat is ignored. +// +// `window.dispatch_action` runs the action with no keystroke behind it, so the keystroke observer +// never sees it — exactly the palette's case. +// +// Hostile halves, the wrong collapse first: a keyless start is not a press still held — time +// passing and a stray key-up of the shortcut's key do not stop it as a hold, and the next press +// (keyless or keyed) is not taken for a key repeat; then the wrong split: a keyed tap and a keyed +// hold still behave as before, including after a keyless toggle; then the plain case. + +use std::time::Duration; + +use gpui::{TestAppContext, VisualTestContext}; +use voice_input::{ToggleRecording, ViewState, VoiceInput}; + +use crate::voice_keys_tests::{ + REMOTE_SETTINGS, Thread, answering, counts, focused_thread, init_voice, press, release, +}; + +/// Longer than the 300 ms tap window. +const PAST_THE_TAP_WINDOW: Duration = Duration::from_millis(400); + +fn keyless(vcx: &mut VisualTestContext) { + vcx.dispatch_action(ToggleRecording); + vcx.run_until_parked(); +} + +fn recording(thread: &Thread, cx: &TestAppContext) -> bool { + let owner = thread.view(cx).entity_id(); + cx.read(|cx| { + matches!( + VoiceInput::global(cx).read(cx).view_state(owner), + ViewState::Recording { .. } + ) + }) +} + +// ── the wrong collapse: a keyless start is no held press ─────────────────── + +#[gpui::test] +async fn voice_keyless_start_is_a_toggle_that_time_and_a_key_up_do_not_stop( + cx: &mut TestAppContext, +) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (thread, mut vcx) = focused_thread(cx).await; + + keyless(&mut vcx); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "R2.7 a keyless toggle starts recording" + ); + + std::thread::sleep(PAST_THE_TAP_WINDOW); + vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (1, 0, 0), "still recording past 300 ms"); + std::thread::sleep(Duration::from_millis(1_600)); + vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (1, 0, 0), "still recording at 2 s"); + + // A key-up of the shortcut's key, with no press behind it: a toggle waits for no release, so + // it cannot be one. + release(&mut vcx, "v"); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "R2.7 never waits for a key release: a stray key-up does not end it as a hold" + ); + assert!(recording(&thread, cx), "the toggle recording runs on"); +} + +#[gpui::test] +async fn voice_keyless_second_dispatch_stops_it(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + keyless(&mut vcx); + assert_eq!(counts(&voice.mic), (1, 0, 0)); + keyless(&mut vcx); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "R2.3/R2.7 the second keyless toggle stops and transcribes: it is not a key repeat" + ); +} + +#[gpui::test] +async fn voice_keyless_start_then_a_keyed_tap_stops_it(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + keyless(&mut vcx); + assert_eq!(counts(&voice.mic), (1, 0, 0)); + // Straight away, inside 300 ms of the keyless start. + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "R2.3/R3.2 the shortcut pressed after a keyless start stops it: not taken for a repeat" + ); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (1, 1, 0), "its release changes nothing"); +} + +// ── the wrong split: the keyed paths are unchanged ───────────────────────── + +#[gpui::test] +async fn voice_keyless_keyed_tap_still_toggles(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "R2.2 a keyed tap keeps recording" + ); + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "R2.3 the next press stops it" + ); +} + +#[gpui::test] +async fn voice_keyless_keyed_hold_still_stops_on_release(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + press(&mut vcx); // a key repeat while held + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + assert_eq!(counts(&voice.mic), (1, 0, 0), "R3.2 the repeat is ignored"); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "R3.1 the release ends the hold" + ); +} + +#[gpui::test] +async fn voice_keyless_toggle_leaves_the_keyed_hold_working(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + keyless(&mut vcx); + keyless(&mut vcx); + // Lets the transcription finish, so the next press can start again. + vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (1, 1, 0), "a keyless start and stop"); + + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (2, 1, 0), + "a fresh press after a keyless toggle starts: nothing was left held" + ); + std::thread::sleep(PAST_THE_TAP_WINDOW); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (2, 2, 0), "R3.1 and is a hold"); +} diff --git a/crates/agent_ui/src/voice_keymap_v3_tests.rs b/crates/agent_ui/src/voice_keymap_v3_tests.rs new file mode 100644 index 0000000..82a742c --- /dev/null +++ b/crates/agent_ui/src/voice_keymap_v3_tests.rs @@ -0,0 +1,364 @@ +// Story 044, task 8.2 (refine v3) — Escape yields to an open completion menu, and vim binds the +// voice toggle per OS (R2.2, R2.5, R6.7; design.md C3 "As built (v3)"). Wired in the crate root +// `src/agent_ui.rs` beside `voice_keys_tests`, whose harness it reuses: +// +// #[cfg(test)] +// mod voice_keymap_v3_tests; +// +// Validation filter: `voice_keymap_v3` (every test path is `voice_keymap_v3_tests::…`). +// +// Contract: with a completion menu open in the composer, Escape resolves as it would with no +// voice state at all (the editor's own Escape closes the menu), and the recording or the consent +// stays; with no menu, Escape in the composer is `voice_input::CancelRecording` while recording and +// `voice_input::DismissDownload` while the consent is asked. `vim.json` binds ToggleRecording +// twice: `cmd-alt-v` under `AcpThread && os == macos`, `ctrl-alt-v` under `AcpThread && os != +// macos`. The thread view's `AcpThread` key context carries `os`. +// +// The resolution tests build the context stack by hand — [Workspace, AcpThread (+ voice key), +// Editor (+ voice key, + menu and showing_completions when the menu shows)] — and resolve +// `escape` against each keymap (vim.json layered over default-linux, as Zed loads it). The +// expected action with the menu open is read from the same keymap with the voice keys removed, +// so the test does not hard-code which editor action closes a menu in each keymap. +// +// Hostile halves, the wrong collapse first: the voice blocks do not swallow the Escape that closes +// a menu (the first Escape cancels nothing), and on Linux vim does not bind `cmd-alt-v`; then the +// wrong split: with no menu, Escape still cancels the recording and declines the consent, and the +// second Escape (menu closed) cancels; then the plain case. + +use gpui::{KeyBinding, KeyContext, Keymap, Keystroke, TestAppContext, VisualTestContext}; +use voice_input::ToggleRecording; + +use crate::test_support::init_test; +use crate::voice_keys_tests::{ + REMOTE_SETTINGS, Thread, answering, counts, focused_thread, init_voice, press, release, +}; + +const DEFAULT_KEYMAPS: [&str; 3] = [ + "keymaps/default-linux.json", + "keymaps/default-windows.json", + "keymaps/default-macos.json", +]; +const VIM: &str = "keymaps/vim.json"; + +const CANCEL_RECORDING: &str = "voice_input::CancelRecording"; +const DISMISS_DOWNLOAD: &str = "voice_input::DismissDownload"; + +fn load(asset: &str, cx: &mut TestAppContext) -> Vec { + cx.update(|cx| { + settings::KeymapFile::load_asset_allow_partial_failure(asset, cx).expect("keymap loads") + }) +} + +/// Every keymap under test, vim.json on top of the Linux default as Zed loads it. +fn keymaps(cx: &mut TestAppContext) -> Vec<(String, Keymap)> { + let mut keymaps: Vec<(String, Keymap)> = DEFAULT_KEYMAPS + .iter() + .map(|name| (name.to_string(), Keymap::new(load(name, cx)))) + .collect(); + let mut vim = load("keymaps/default-linux.json", cx); + vim.extend(load(VIM, cx)); + keymaps.push((format!("{VIM} over default-linux"), Keymap::new(vim))); + keymaps +} + +#[derive(Clone, Copy)] +struct Stack { + voice: Option<&'static str>, + menu: bool, + vim_insert: bool, +} + +// Linux on every host: the stacks are resolved against default-linux.json and vim.json's +// `os != macos` arm, so the os they carry is part of the fixture, not the host's. +fn stack(spec: Stack) -> Vec { + let mut workspace = KeyContext::new_with_defaults(gpui::OperatingSystem::Linux); + workspace.add("Workspace"); + let mut thread = KeyContext::new_with_defaults(gpui::OperatingSystem::Linux); + thread.add("AcpThread"); + let mut editor = KeyContext::new_with_defaults(gpui::OperatingSystem::Linux); + editor.add("Editor"); + editor.set("mode", "auto_height"); + if let Some(voice) = spec.voice { + thread.add(voice); + editor.add(voice); + } + if spec.menu { + editor.add("menu"); + editor.add("showing_completions"); + } + if spec.vim_insert { + editor.add("VimControl"); + editor.set("vim_mode", "insert"); + } + vec![workspace, thread, editor] +} + +fn escape_in(keymap: &Keymap, spec: Stack) -> Option { + let escape = Keystroke::parse("escape").expect("keystroke"); + keymap + .bindings_for_input(&[escape], &stack(spec)) + .0 + .first() + .map(|binding| binding.action().name().to_string()) +} + +// ── the wrong collapse: an open menu takes the first Escape ──────────────── + +#[gpui::test] +fn voice_keymap_v3_escape_with_a_menu_open_resolves_as_without_voice(cx: &mut TestAppContext) { + init_test(cx); + for (name, keymap) in keymaps(cx) { + for vim_insert in [false, true] { + let without_voice = escape_in( + &keymap, + Stack { + voice: None, + menu: true, + vim_insert, + }, + ); + assert!( + without_voice + .as_deref() + .is_some_and(|action| !action.starts_with("voice_input::")), + "{name} (vim insert: {vim_insert}): precondition, Escape over a menu is the editor's: {without_voice:?}" + ); + for voice in ["voice_recording", "voice_consent"] { + let with_voice = escape_in( + &keymap, + Stack { + voice: Some(voice), + menu: true, + vim_insert, + }, + ); + assert_eq!( + with_voice, without_voice, + "{name} (vim insert: {vim_insert}): R2.5 under {voice} with a completion menu open, \ + the first Escape closes the menu" + ); + } + } + } +} + +#[gpui::test] +fn voice_keymap_v3_default_keymaps_close_the_menu_with_editor_cancel(cx: &mut TestAppContext) { + init_test(cx); + for (name, keymap) in keymaps(cx) { + let action = escape_in( + &keymap, + Stack { + voice: Some("voice_recording"), + menu: true, + vim_insert: false, + }, + ); + assert_eq!( + action.as_deref(), + Some("editor::Cancel"), + "{name}: Escape over the composer's completion menu is editor::Cancel while recording" + ); + } +} + +// ── the wrong split: with no menu, Escape is the voice strip's ───────────── + +#[gpui::test] +fn voice_keymap_v3_escape_without_a_menu_cancels_or_declines(cx: &mut TestAppContext) { + init_test(cx); + for (name, keymap) in keymaps(cx) { + for vim_insert in [false, true] { + for (voice, expected) in [ + ("voice_recording", CANCEL_RECORDING), + ("voice_consent", DISMISS_DOWNLOAD), + ] { + let action = escape_in( + &keymap, + Stack { + voice: Some(voice), + menu: false, + vim_insert, + }, + ); + assert_eq!( + action.as_deref(), + Some(expected), + "{name} (vim insert: {vim_insert}): R2.5/R6.7 Escape under {voice} with no menu" + ); + } + } + } +} + +// ── vim binds the toggle per OS ───────────────────────────────────────────── + +fn toggle_blocks(content: &str) -> Vec<(String, String)> { + let sections = + settings::parse_json_with_comments::>(content).expect("keymap JSON"); + let mut found = Vec::new(); + for section in sections { + let context = section + .get("context") + .and_then(|context| context.as_str()) + .unwrap_or("") + .split_whitespace() + .collect::>() + .join(" "); + let Some(bindings) = section.get("bindings").and_then(|b| b.as_object()) else { + continue; + }; + for (keys, action) in bindings { + if action.as_str() == Some("voice_input::ToggleRecording") { + found.push((context.clone(), keys.clone())); + } + } + } + found +} + +#[test] +fn voice_keymap_v3_vim_binds_the_toggle_once_per_os() { + let found = toggle_blocks(include_str!("../../../assets/keymaps/vim.json")); + assert_eq!( + found.len(), + 2, + "vim.json binds ToggleRecording exactly twice: {found:?}" + ); + assert!( + found.iter().any(|(context, keys)| keys == "cmd-alt-v" + && context.contains("AcpThread") + && context.contains("os == macos")), + "cmd-alt-v under AcpThread && os == macos: {found:?}" + ); + assert!( + found.iter().any(|(context, keys)| keys == "ctrl-alt-v" + && context.contains("AcpThread") + && context.contains("os != macos")), + "ctrl-alt-v under AcpThread && os != macos: {found:?}" + ); +} + +fn is_chord(binding: &KeyBinding, chord: &'static str) -> bool { + let chord = Keystroke::parse(chord).expect("keystroke"); + binding.keystrokes().len() == 1 + && binding.keystrokes()[0].inner().modifiers == chord.modifiers + && binding.keystrokes()[0].inner().key == chord.key +} + +#[gpui::test] +async fn voice_keymap_v3_thread_context_carries_the_os(cx: &mut TestAppContext) { + let _voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + // The host os the app reports, which is what the thread's context is built from. + let expected = vcx.update(|_, cx| { + KeyContext::new_with_defaults(cx.platform_info().os) + .get("os") + .cloned() + }); + let thread_os = vcx.update(|window, _| { + window + .context_stack() + .into_iter() + .find(|context| context.contains("AcpThread")) + .map(|context| context.get("os").cloned()) + }); + assert_eq!( + thread_os, + Some(expected), + "the AcpThread key context names the os, so `AcpThread && os == …` can match" + ); +} + +#[gpui::test] +async fn voice_keymap_v3_vim_on_this_os_binds_only_its_own_chord(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + cx.update(|cx| { + let vim = settings::KeymapFile::load_asset_allow_partial_failure(VIM, cx) + .expect("vim keymap loads"); + cx.bind_keys(vim); + }); + let (thread, mut vcx) = focused_thread(cx).await; + let focus = composer_focus(&thread, &mut vcx); + let bindings = vcx.update(|window, _| window.bindings_for_action_in(&ToggleRecording, &focus)); + let (own, other) = if cfg!(target_os = "macos") { + ("cmd-alt-v", "ctrl-alt-v") + } else { + ("ctrl-alt-v", "cmd-alt-v") + }; + assert!( + bindings.iter().any(|binding| is_chord(binding, own)), + "{own} is bound in the composer with vim.json loaded" + ); + assert!( + !bindings.iter().any(|binding| is_chord(binding, other)), + "{other} is not bound on this OS with vim.json loaded" + ); + if !cfg!(target_os = "macos") { + press(&mut vcx); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "R2.2 the vim user's ctrl-alt-v records" + ); + } +} + +fn composer_focus(thread: &Thread, vcx: &mut VisualTestContext) -> gpui::FocusHandle { + let editor = thread.editor(vcx); + editor.read_with(vcx, |editor, cx| gpui::Focusable::focus_handle(editor, cx)) +} + +// ── a live completion menu in a recording thread ─────────────────────────── + +#[gpui::test] +async fn voice_keymap_v3_first_escape_closes_the_menu_and_the_second_cancels( + cx: &mut TestAppContext, +) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "precondition: a toggle recording runs" + ); + + // `@` opens the composer's mention menu. + vcx.simulate_input("@"); + vcx.run_until_parked(); + let message_editor = thread.editor(&vcx); + let menu_open = |vcx: &mut VisualTestContext| { + message_editor.read_with(vcx, |message_editor, cx| { + message_editor + .editor() + .read(cx) + .has_visible_completions_menu() + }) + }; + assert!( + menu_open(&mut vcx), + "precondition: the harness opened the completion menu" + ); + + vcx.simulate_keystrokes("escape"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "R2.5 the first Escape closes the menu: the recording goes on, nothing cancelled" + ); + assert!(!menu_open(&mut vcx), "the first Escape closed the menu"); + + vcx.simulate_keystrokes("escape"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 1), + "R2.5 with the menu closed, Escape cancels the recording" + ); +} diff --git a/crates/agent_ui/src/voice_keys_fix_tests.rs b/crates/agent_ui/src/voice_keys_fix_tests.rs new file mode 100644 index 0000000..f992177 --- /dev/null +++ b/crates/agent_ui/src/voice_keys_fix_tests.rs @@ -0,0 +1,381 @@ +// Story 044, task 5.3 fix round 1 — the three findings of the 5.3 tech review (R2.3, R2.5, R3.1, +// R3.2). Wired in the crate root `src/agent_ui.rs` beside `voice_keys_tests`, whose harness it +// reuses. Validation filter: `voice_keys_fix` (every test path is `voice_keys_fix_tests::…`). +// +// M1: Escape is claimed by the voice strip only while the consent is asked, never while the +// model downloads: during the download it keeps its usual meaning and the download goes on. +// M2: on macOS the key-up of `v` is not delivered while Cmd is held, so letting go of the +// shortcut's modifiers is a release too; a key-up arriving afterwards changes nothing. +// L1: the release is matched against every binding of `ToggleRecording`, not only the binding of +// highest precedence. + +use std::{sync::Arc, time::Duration}; + +use gpui::{ + Focusable as _, KeyBinding, KeyDownEvent, Keystroke, Modifiers, TestAppContext, + VisualTestContext, +}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use voice_input::state::Input; +use voice_input::{DismissDownload, ToggleRecording, ViewState, VoiceInput}; + +use crate::voice_button::send; +use crate::voice_keys_tests::{ + REMOTE_SETTINGS, Thread, answering, counts, focused_thread, init_voice, press, release, +}; + +/// Longer than the 300 ms tap window, so a release past it ends a push-to-talk hold (R3.1). +const PAST_THE_TAP_WINDOW: Duration = Duration::from_millis(400); + +/// Local mode with no model on disk: the first press asks for the download (R6.1). +const LOCAL_SETTINGS: &str = r#"{ "voice_input": { "model": "tiny" } }"#; + +/// An HTTP client whose every request stays in flight: a download started with it keeps running +/// for as long as the test needs. +fn never_answering() -> Arc { + FakeHttpClient::create(|_req| futures::future::pending::>>()) +} + +fn view_state(thread: &Thread, cx: &TestAppContext) -> ViewState { + let owner = thread.view(cx).entity_id(); + cx.read(|cx| VoiceInput::global(cx).read(cx).view_state(owner)) +} + +fn composer_names_voice_consent(vcx: &mut VisualTestContext) -> bool { + vcx.update(|window, _| { + window + .context_stack() + .iter() + .any(|context| context.contains("voice_consent")) + }) +} + +fn tap(vcx: &mut VisualTestContext) { + press(vcx); + release(vcx, "ctrl-alt-v"); + vcx.run_until_parked(); +} + +// ── M1: Escape and the model download ─────────────────────────────────────── + +#[gpui::test] +async fn voice_keys_fix_escape_declines_the_consent_but_leaves_a_download_running( + cx: &mut TestAppContext, +) { + let voice = init_voice(LOCAL_SETTINGS, never_answering(), cx); + let (thread, mut vcx) = focused_thread(cx).await; + + // The consent the user just triggered: Escape declines it. + tap(&mut vcx); + assert!( + matches!(view_state(&thread, cx), ViewState::AwaitingConsent { .. }), + "a press with no model asks for the download: {:?}", + view_state(&thread, cx) + ); + assert!( + composer_names_voice_consent(&mut vcx), + "the composer names voice_consent while the consent is asked" + ); + vcx.simulate_keystrokes("escape"); + vcx.run_until_parked(); + assert!( + matches!(view_state(&thread, cx), ViewState::Idle { .. }), + "Escape declines the consent: {:?}", + view_state(&thread, cx) + ); + + // The download, once consented to, runs in the background while the user keeps typing. + tap(&mut vcx); + assert!( + matches!(view_state(&thread, cx), ViewState::AwaitingConsent { .. }), + "{:?}", + view_state(&thread, cx) + ); + vcx.update(|_, cx| send(Input::ConsentGiven, cx)); + vcx.run_until_parked(); + assert!( + matches!(view_state(&thread, cx), ViewState::Downloading { .. }), + "the consent starts the download: {:?}", + view_state(&thread, cx) + ); + assert!( + !composer_names_voice_consent(&mut vcx), + "the composer does not name voice_consent while the model downloads" + ); + let editor_focus = thread + .editor(cx) + .read_with(&vcx, |editor, cx| editor.focus_handle(cx)); + let dismiss_bindings = vcx.update(|window, _| { + window + .bindings_for_action_in(&DismissDownload, &editor_focus) + .len() + }); + assert_eq!( + dismiss_bindings, 0, + "no key reaches DismissDownload from the composer while the model downloads" + ); + vcx.simulate_keystrokes("escape"); + vcx.run_until_parked(); + assert!( + matches!(view_state(&thread, cx), ViewState::Downloading { .. }), + "Escape in the composer keeps its usual meaning: the download is not cancelled: {:?}", + view_state(&thread, cx) + ); + assert_eq!( + counts(&voice.mic), + (0, 0, 0), + "no recording was ever involved" + ); +} + +// ── M2: letting go of the modifiers releases the shortcut ─────────────────── + +#[gpui::test] +async fn voice_keys_fix_releasing_the_modifiers_ends_a_hold_without_a_key_up( + cx: &mut TestAppContext, +) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + + // A hold: `v` is let go of while Cmd is down, so its key-up never comes (macOS). + press(&mut vcx); + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + vcx.simulate_modifiers_change(Modifiers::none()); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "R3.1 releasing the shortcut's modifiers ends the hold" + ); + + // The key-up arriving afterwards is the same release, seen twice: nothing more happens. + release(&mut vcx, "v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "a key-up after the modifiers release neither starts nor stops anything" + ); + + // A tap released by the modifiers alone is a release too: the next press is a fresh one. + press(&mut vcx); + vcx.run_until_parked(); + vcx.simulate_modifiers_change(Modifiers::none()); + release(&mut vcx, "v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (2, 1, 0), + "R2.2 a tap keeps recording after its release" + ); + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (2, 2, 0), + "R2.3/R3.2 the press after a modifiers release is not taken for a repeat: it stops" + ); +} + +#[gpui::test] +async fn voice_keys_fix_a_modifier_still_held_does_not_release(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + // Ctrl let go of, Alt still down, `v` still down: the shortcut is still held. + vcx.simulate_modifiers_change(Modifiers::alt()); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "one of the shortcut's modifiers is still held" + ); + vcx.simulate_modifiers_change(Modifiers::none()); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "the last of them let go of releases the hold" + ); +} + +// ── L1: every binding of the shortcut releases ────────────────────────────── + +#[gpui::test] +async fn voice_keys_fix_every_binding_of_the_shortcut_releases(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + // A user's own second binding, on a different final key. User keymaps load after the + // defaults, so it takes precedence over `ctrl-alt-v`. Not `f9`: the default keymap binds it + // to `editor::ToggleBreakpoint` in the composer's deeper `Editor` context. + cx.update(|cx| { + cx.bind_keys([KeyBinding::new( + "ctrl-alt-m", + ToggleRecording, + Some("AcpThread"), + )]); + }); + let (_thread, mut vcx) = focused_thread(cx).await; + + vcx.simulate_keystrokes("ctrl-alt-m"); + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + release(&mut vcx, "ctrl-alt-m"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "R3.1 the release of ctrl-alt-m ends its hold" + ); + // Lets the transcription finish, so the next press starts again. + vcx.run_until_parked(); + + press(&mut vcx); + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (2, 2, 0), + "R3.1 the release of v ends a ctrl-alt-v hold, though ctrl-alt-m is the binding shown" + ); +} + +// ── Fix round 1b ───────────────────────────────────────────────────────────── +// +// M-1: gpui's Wayland client repeats a held key on the client side by replaying the keystroke it +// captured at the press, `is_held: true`, until that key's key-up; letting go of Ctrl and Alt +// changes neither. After a release through the modifiers, those replays are not fresh presses. + +fn stale_repeat(vcx: &mut VisualTestContext) { + vcx.simulate_event(KeyDownEvent { + keystroke: Keystroke::parse("ctrl-alt-v").expect("keystroke"), + is_held: true, + prefer_character_input: false, + }); +} + +#[gpui::test] +async fn voice_keys_fix_repeats_after_a_modifiers_release_start_nothing(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + vcx.simulate_modifiers_change(Modifiers::none()); + // Lets the transcription finish, so a press read as fresh would start a new recording. + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "the modifiers release ended the hold" + ); + + stale_repeat(&mut vcx); + stale_repeat(&mut vcx); + vcx.run_until_parked(); + release(&mut vcx, "v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "a key still repeating after its modifiers were let go of starts no recording" + ); + + // The key-up ended the stale repeat: the next press is a fresh one again. + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic).0, + 2, + "a press after the key-up starts recording" + ); +} + +#[gpui::test] +async fn voice_keys_fix_a_chord_pressed_again_after_a_modifiers_release_counts( + cx: &mut TestAppContext, +) { + // macOS: no key-up of `v` ever comes while Cmd is held, so pressing a modifier of the chord + // again is what makes the next press a fresh one. + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + vcx.simulate_modifiers_change(Modifiers::none()); + vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (1, 1, 0)); + + vcx.simulate_modifiers_change(Modifiers::control()); + vcx.simulate_modifiers_change(Modifiers::control() | Modifiers::alt()); + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic).0, + 2, + "the chord's modifiers pressed again make the next press a fresh one" + ); +} + +// L-3: the order users usually type — `v` up inside the tap window, the modifiers let go of later. + +#[gpui::test] +async fn voice_keys_fix_a_tap_is_not_ended_by_a_later_modifiers_release(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + std::thread::sleep(PAST_THE_TAP_WINDOW); + vcx.simulate_modifiers_change(Modifiers::none()); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "R2.2 the tap keeps recording: the later modifiers release is not a second release" + ); + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "R2.3 the next press stops the toggle" + ); +} + +// L-1: the keystroke observer is app-wide. A press without a key (the command palette) leaves a +// view waiting for its keystroke; a chord pressed in another window must not be recorded as its. + +#[gpui::test] +async fn voice_keys_fix_a_press_in_another_window_is_not_recorded_as_this_views( + cx: &mut TestAppContext, +) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_other, mut other_vcx) = focused_thread(cx).await; + let (_thread, mut vcx) = focused_thread(cx).await; + + // No keystroke: the command palette's way in. This view's recording starts and waits. + vcx.dispatch_action(ToggleRecording); + vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (1, 0, 0)); + + // The chord in the other window: the machine takes it for a repeat of the held press. + press(&mut other_vcx); + other_vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (1, 0, 0)); + + // Shift alone in this window holds none of that chord's modifiers. Had this view recorded the + // other window's chord as its own, this would be its release, and end the recording. + std::thread::sleep(PAST_THE_TAP_WINDOW); + vcx.simulate_modifiers_change(Modifiers::shift()); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "a chord pressed in another window is not this view's to release" + ); +} diff --git a/crates/agent_ui/src/voice_keys_tests.rs b/crates/agent_ui/src/voice_keys_tests.rs new file mode 100644 index 0000000..b063f1d --- /dev/null +++ b/crates/agent_ui/src/voice_keys_tests.rs @@ -0,0 +1,475 @@ +// Story 044, task 5.3 — shortcut press, release, Escape and focus loss in the composer (R1.2, R2.2, +// R2.3, R2.5, R3.2, R3.3; design.md C2, C3). Wired in the crate root `src/agent_ui.rs` as +// +// #[cfg(test)] +// mod voice_keys_tests; +// +// Validation filter: `voice_keys` (every test path is `voice_keys_tests::…`). The harness below is +// `pub(crate)` because `voice_events_tests` reuses it. +// +// Contract: design.md C3 (the two bindings in all four keymaps), C2 (`ToggleRecording` → ShortcutDown, +// key-up of the binding's non-modifier key → ShortcutUp, `CancelRecording` → Cancel, window +// deactivation → WindowDeactivated, `voice_recording` key context while recording). +// Assumed voice_input seams (see voice_input/src/entity_tests.rs): `voice_input::init_in(http, +// models_dir, capture, cx)`, `voice_input::capture::{CaptureBackend, ActiveCapture, CaptureEvent, +// Recording, Ended}`. The thread view is built like `conversation_view::tests:: +// setup_conversation_view` (private there), as the root of its own window; on the series tip, copy +// its current `ConversationView::new` arguments if they moved. Remote mode on a loopback URL, so no +// consent strip and no model are involved; the microphone is always the fake. +// +// Hostile halves: key-down repeats while held are ignored AND a fresh press after the release stops +// the toggle; the release of an unrelated key does not count AND the release of `v` with whatever +// modifiers are still down does; Escape cancels while recording AND is not claimed while idle (the +// binding is only live under `voice_recording`); focus loss stops a held recording AND leaves a +// toggle running; the shortcut does nothing when the setting is off. + +use std::{ + path::PathBuf, + rc::Rc, + sync::{ + Arc, Mutex, + atomic::{AtomicUsize, Ordering}, + }, +}; + +use agent::ThreadStore; +use fs::FakeFs; +use gpui::{ + AnyWindowHandle, AppContext as _, Entity, KeyUpEvent, Keystroke, TestAppContext, + UpdateGlobal as _, VisualContext as _, VisualTestContext, +}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use project::Project; +use settings::SettingsStore; +use voice_input::capture::{ActiveCapture, CaptureBackend, CaptureEventSender, Ended, Recording}; +use voice_input::{CancelRecording, ToggleRecording}; +use workspace::{MultiWorkspace, Workspace}; + +use crate::agent_connection_store::AgentConnectionStore; +use crate::conversation_view::ConversationView; +use crate::conversation_view::thread_view::ThreadView; +use crate::message_editor::MessageEditor; +use crate::test_support::{StubAgentServer, init_test}; +use crate::{Agent, AgentThreadSource}; + +// ── shared harness ────────────────────────────────────────────────────────── + +#[derive(Default)] +pub(crate) struct FakeMic { + pub starts: AtomicUsize, + pub stops: AtomicUsize, + pub cancels: AtomicUsize, + pub events: Mutex>, +} + +struct Backend(Arc); +struct Active(Arc); + +impl CaptureBackend for Backend { + fn start( + &self, + _device: Option, + events: CaptureEventSender, + ) -> anyhow::Result> { + self.0.starts.fetch_add(1, Ordering::SeqCst); + *self.0.events.lock().expect("lock") = Some(events); + Ok(Box::new(Active(self.0.clone()))) + } +} + +impl ActiveCapture for Active { + fn stop(self: Box) -> Recording { + self.0.stops.fetch_add(1, Ordering::SeqCst); + let samples = (0..48_000).map(|i| 0.2 * (i as f32 * 0.05).sin()).collect(); + Recording { + samples, + sample_rate: 48_000, + ended: Ended::User, + } + } + fn cancel(self: Box) { + self.0.cancels.fetch_add(1, Ordering::SeqCst); + } +} + +pub(crate) const REMOTE_SETTINGS: &str = r#"{ "voice_input": { "remote": { "enabled": true, + "url": "http://127.0.0.1:9/v1/audio/transcriptions", "api_key_env": "ZEO_VOICE_TEST_UNSET_044_9F3C" } } }"#; + +pub(crate) struct Voice { + pub mic: Arc, + pub _models: tempfile::TempDir, +} + +/// init_test + the default Linux keymap + user settings + `voice_input::init_in` with fakes + +/// `agent_ui::init`'s voice notices. +pub(crate) fn init_voice( + settings_json: &str, + http: Arc, + cx: &mut TestAppContext, +) -> Voice { + init_test(cx); + cx.executor().allow_parking(); + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(settings_json, cx) + .result() + .expect("settings parse"); + }); + let bindings = settings::KeymapFile::load_asset_allow_partial_failure( + "keymaps/default-linux.json", + cx, + ) + .expect("the default keymap loads"); + cx.bind_keys(bindings); + }); + let mic = Arc::new(FakeMic::default()); + let models = tempfile::tempdir().expect("tempdir"); + let backend: Arc = Arc::new(Backend(mic.clone())); + let dir: PathBuf = models.path().join("voice").join("models"); + cx.update(|cx| { + voice_input::init_in(http, dir, backend, cx); + // What `agent_ui::init` adds for voice input, in production's order (after the entity). + crate::voice_notice::init(cx); + }); + Voice { + mic, + _models: models, + } +} + +pub(crate) fn answering(text_json: &'static str) -> Arc { + FakeHttpClient::create(move |_req| async move { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from(text_json))?) + }) +} + +pub(crate) struct Thread { + pub window: AnyWindowHandle, + pub conversation: Entity, + pub workspace: Entity, +} + +impl Thread { + pub fn view(&self, cx: &TestAppContext) -> Entity { + cx.read(|cx| { + self.conversation + .read(cx) + .active_thread() + .expect("an active thread") + .clone() + }) + } + pub fn editor(&self, cx: &TestAppContext) -> Entity { + let view = self.view(cx); + cx.read(|cx| view.read(cx).message_editor.clone()) + } + pub fn vcx(&self, cx: &mut TestAppContext) -> VisualTestContext { + VisualTestContext::from_window(self.window, cx) + } +} + +/// A thread view as the root of its own window, sharing one workspace window per call. +pub(crate) async fn open_thread(cx: &mut TestAppContext) -> Thread { + let fs = FakeFs::new(cx.executor()); + let project = Project::test(fs, [], cx).await; + let mw = cx.add_window(|window, cx| MultiWorkspace::test_new(project.clone(), window, cx)); + let workspace = mw + .read_with(cx, |mw, _| mw.workspace().clone()) + .expect("workspace"); + let thread_store = cx.new(|cx| ThreadStore::new(cx)); + let connection_store = cx.new(|cx| AgentConnectionStore::new(project.clone(), cx)); + let window = cx.add_window(|window, cx| { + ConversationView::new( + Rc::new(StubAgentServer::default_response()), + connection_store, + Agent::Custom { id: "Test".into() }, + None, + None, + None, + None, + None, + workspace.downgrade(), + project, + Some(thread_store), + AgentThreadSource::AgentPanel, + window, + cx, + ) + }); + let conversation = window.root(cx).expect("root view"); + cx.run_until_parked(); + Thread { + window: window.into(), + conversation, + workspace, + } +} + +pub(crate) async fn focused_thread(cx: &mut TestAppContext) -> (Thread, VisualTestContext) { + let thread = open_thread(cx).await; + let editor = thread.editor(cx); + let mut vcx = thread.vcx(cx); + // A focused composer sits in the active window; the test platform activates none by itself. + vcx.update(|window, _| window.activate_window()); + vcx.focus(&editor); + vcx.run_until_parked(); + (thread, vcx) +} + +pub(crate) fn press(vcx: &mut VisualTestContext) { + vcx.simulate_keystrokes("ctrl-alt-v"); // key-down only: the key stays "held" +} + +pub(crate) fn release(vcx: &mut VisualTestContext, keystroke: &'static str) { + vcx.simulate_event(KeyUpEvent { + keystroke: Keystroke::parse(keystroke).expect("keystroke"), + }); +} + +pub(crate) fn counts(mic: &FakeMic) -> (usize, usize, usize) { + ( + mic.starts.load(Ordering::SeqCst), + mic.stops.load(Ordering::SeqCst), + mic.cancels.load(Ordering::SeqCst), + ) +} + +// ── keymaps (C3; also 6.1's "every action name resolves") ─────────────────── + +const KEYMAPS: [(&str, &str, &str); 4] = [ + ( + "keymaps/default-linux.json", + include_str!("../../../assets/keymaps/default-linux.json"), + "ctrl-alt-v", + ), + ( + "keymaps/default-windows.json", + include_str!("../../../assets/keymaps/default-windows.json"), + "ctrl-alt-v", + ), + ( + "keymaps/default-macos.json", + include_str!("../../../assets/keymaps/default-macos.json"), + "cmd-alt-v", + ), + ( + "keymaps/vim.json", + include_str!("../../../assets/keymaps/vim.json"), + "", + ), +]; + +fn sections(content: &str) -> Vec { + settings::parse_json_with_comments::>(content).expect("keymap JSON") +} + +fn bound(content: &str, action: &str) -> Vec<(String, String)> { + let mut found = Vec::new(); + for section in sections(content) { + let context = section + .get("context") + .and_then(|c| c.as_str()) + .unwrap_or("") + .to_string(); + if let Some(bindings) = section.get("bindings").and_then(|b| b.as_object()) { + for (keys, value) in bindings { + if value.as_str() == Some(action) { + found.push((context.clone(), keys.clone())); + } + } + } + } + found +} + +#[test] +fn voice_keys_all_four_keymaps_bind_toggle_in_acp_thread() { + for (name, content, key) in KEYMAPS { + let found = bound(content, "voice_input::ToggleRecording"); + assert!( + found.iter().any(|(ctx, keys)| ctx.contains("AcpThread") + && (keys == key + || (key.is_empty() && (keys == "ctrl-alt-v" || keys == "cmd-alt-v")))), + "{name}: ToggleRecording on {key:?} in AcpThread, found {found:?}" + ); + } +} + +#[test] +fn voice_keys_escape_cancels_only_under_voice_recording() { + for (name, content, _) in KEYMAPS { + let found = bound(content, "voice_input::CancelRecording"); + assert!( + found + .iter() + .any(|(ctx, keys)| keys == "escape" && ctx.contains("voice_recording")), + "{name}: escape → CancelRecording under voice_recording, found {found:?}" + ); + assert!( + found.iter().all(|(ctx, _)| ctx.contains("voice_recording")), + "{name}: never an unconditional Escape: {found:?}" + ); + } +} + +#[gpui::test] +fn voice_keys_action_names_resolve_in_every_keymap(cx: &mut TestAppContext) { + init_test(cx); + for (name, _, _) in KEYMAPS { + let names: Vec = cx.update(|cx| { + settings::KeymapFile::load_asset_allow_partial_failure(name, cx) + .expect("loads") + .iter() + .map(|b| b.action().name().to_string()) + .collect() + }); + for action in [ + "voice_input::ToggleRecording", + "voice_input::CancelRecording", + ] { + assert!( + names.iter().any(|n| n == action), + "{name}: {action} resolves to an action" + ); + } + } +} + +// ── key handling in a real thread view ────────────────────────────────────── + +#[gpui::test] +async fn voice_keys_tap_toggles_and_the_next_press_stops(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "R2.2 a tap keeps recording after the release" + ); + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic).1, + 1, + "R2.3 a fresh press stops the toggle" + ); +} + +#[gpui::test] +async fn voice_keys_repeats_while_held_are_ignored(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + for _ in 0..4 { + press(&mut vcx); + } + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "R3.2 one start, no stop from repeats" + ); +} + +#[gpui::test] +async fn voice_keys_unrelated_key_up_does_not_release_the_shortcut(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + release(&mut vcx, "x"); + press(&mut vcx); // still held as far as the machine knows: a repeat + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "an unrelated key-up is not the shortcut's release" + ); +} + +#[gpui::test] +async fn voice_keys_release_of_v_counts_whatever_modifiers_remain(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + release(&mut vcx, "alt-v"); // ctrl let go first; X11 sends no key-up for the modifier itself + press(&mut vcx); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic).1, + 1, + "the release was seen, so the next press stops the toggle" + ); +} + +#[gpui::test] +async fn voice_keys_escape_cancels_while_recording_and_is_free_while_idle(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + let idle_bindings = vcx.update(|window, _| window.bindings_for_action(&CancelRecording).len()); + assert_eq!( + idle_bindings, 0, + "Escape is not claimed while idle (no voice_recording context)" + ); + + press(&mut vcx); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + let live = vcx.update(|window, _| window.bindings_for_action(&CancelRecording).len()); + assert!( + live > 0, + "voice_recording is in the composer's key context while recording" + ); + vcx.simulate_keystrokes("escape"); + vcx.run_until_parked(); + assert_eq!( + counts(&voice.mic), + (1, 0, 1), + "R2.5 cancelled, not stopped: nothing is transcribed" + ); + let after = vcx.update(|window, _| window.bindings_for_action(&CancelRecording).len()); + assert_eq!(after, 0, "the context is gone after the cancel"); +} + +#[gpui::test] +async fn voice_keys_focus_loss_stops_a_hold_but_not_a_toggle(cx: &mut TestAppContext) { + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); // held, not released + vcx.deactivate_window(); + assert_eq!( + counts(&voice.mic), + (1, 1, 0), + "R3.3 a held recording stops and is transcribed" + ); + + let voice = init_voice(REMOTE_SETTINGS, answering(r#"{"text":"ok"}"#), cx); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + release(&mut vcx, "ctrl-alt-v"); + vcx.deactivate_window(); + assert_eq!( + counts(&voice.mic), + (1, 0, 0), + "a toggle keeps recording without focus" + ); +} + +#[gpui::test] +async fn voice_keys_shortcut_does_nothing_when_disabled(cx: &mut TestAppContext) { + let voice = init_voice( + r#"{ "voice_input": { "enabled": false } }"#, + answering("{}"), + cx, + ); + let (_thread, mut vcx) = focused_thread(cx).await; + press(&mut vcx); + release(&mut vcx, "ctrl-alt-v"); + vcx.run_until_parked(); + assert_eq!(counts(&voice.mic), (0, 0, 0), "R1.2 the shortcut is inert"); + let _ = ToggleRecording; // the action type exists even when disabled +} diff --git a/crates/agent_ui/src/voice_notice.rs b/crates/agent_ui/src/voice_notice.rs new file mode 100644 index 0000000..277a0f4 --- /dev/null +++ b/crates/agent_ui/src/voice_notice.rs @@ -0,0 +1,107 @@ +//! What voice input tells the user, as workspace toasts (design.md C1.10, C2; R4.7, R5.1). +//! +//! A composer's own notices go to its thread view's workspace; a transcript whose composer was +//! closed has no view left to show it, so `agent_ui::init` turns it into a toast in the active +//! workspace instead. + +use std::borrow::Cow; + +use gpui::{App, Entity, EntityId, Global, Subscription, WeakEntity}; +use voice_input::{VoiceEvent, VoiceInput}; +use workspace::{MultiWorkspace, Toast, Workspace, notifications::NotificationId}; + +/// The notice for a transcript that arrived after its composer was closed (R4.7). +pub const DISCARDED: &str = "Transcript discarded: the thread was closed"; + +/// Every voice notice shares one id, so a newer notice replaces the previous one in a workspace +/// instead of stacking under it. +struct VoiceToast; + +/// Shows `message` as a toast in `workspace`. A workspace closed under a pending notice loses +/// only that notice, so it is logged as degraded (never with the message, which may quote a +/// transcription error), not as an error. +pub fn show_toast( + workspace: &WeakEntity, + message: impl Into>, + cx: &mut App, +) { + let shown = workspace.update(cx, |workspace, cx| { + workspace.show_toast( + Toast::new(NotificationId::unique::(), message), + cx, + ) + }); + if shown.is_err() { + log::warn!("voice_input event=notice_dropped reason=workspace_released"); + } +} + +/// Subscribes once, app-wide, to the voice input entity and turns `Discarded` into a toast in the +/// active workspace (R4.7). Covers both orders: an entity that already exists, and one created +/// later (a re-initialisation creates a new entity). +pub fn init(cx: &mut App) { + if let Some(voice) = VoiceInput::try_global(cx) { + watch(&voice, cx); + } + cx.observe_new(|_: &mut VoiceInput, _, cx| { + let voice = cx.entity(); + watch(&voice, cx); + }) + .detach(); +} + +/// The one app-wide subscription, to the current voice input entity. +struct DiscardNotices { + voice: EntityId, + _subscription: Subscription, +} + +impl Global for DiscardNotices {} + +/// An entity created in the same update as `init` is seen by both of its paths; the second +/// subscription would show every notice twice. +fn watch(voice: &Entity, cx: &mut App) { + let voice_id = voice.entity_id(); + if cx + .try_global::() + .is_some_and(|notices| notices.voice == voice_id) + { + return; + } + let subscription = cx.subscribe(voice, |_, event, cx| on_voice_event(event, cx)); + cx.set_global(DiscardNotices { + voice: voice_id, + _subscription: subscription, + }); +} + +fn on_voice_event(event: &VoiceEvent, cx: &mut App) { + if let VoiceEvent::Discarded { owner } = event { + notify_discarded(*owner, cx); + } +} + +fn notify_discarded(owner: EntityId, cx: &mut App) { + match active_workspace(cx) { + Some(workspace) => show_toast(&workspace.downgrade(), DISCARDED, cx), + None => log::warn!( + "voice_input event=discard_notice_dropped owner={} reason=no_workspace", + owner.as_u64() + ), + } +} + +/// The workspace of the active window; when the active window holds no workspace (or there is +/// none), the frontmost window that does. +fn active_workspace(cx: &App) -> Option> { + let stack = cx.window_stack().unwrap_or_default(); + cx.active_window() + .into_iter() + .chain(stack) + .chain(cx.windows()) + .find_map(|window| { + let multi_workspace = window.downcast::()?; + let multi_workspace = multi_workspace.read(cx).ok()?; + Some(multi_workspace.workspace().clone()) + }) +} diff --git a/crates/agent_ui/src/voice_refused_log_tests.rs b/crates/agent_ui/src/voice_refused_log_tests.rs new file mode 100644 index 0000000..e91c83e --- /dev/null +++ b/crates/agent_ui/src/voice_refused_log_tests.rs @@ -0,0 +1,194 @@ +// Story 044, task 8.7 (refine v3) — a transcript refused by a read-only composer is logged as +// refused (R4.7, R10.1, R10.3; design.md C2 "v3, refused insert"). Wired in the crate root +// `src/agent_ui.rs` beside `voice_keys_tests`, whose harness it reuses: +// +// #[cfg(test)] +// mod voice_refused_log_tests; +// +// Validation filter: `voice_refused_log` (every test path is `voice_refused_log_tests::…`). +// +// Contract: when `MessageEditor::insert_transcript` refuses (the composer turned read-only while +// the transcript was on its way), the thread view shows its toast as before and calls +// `VoiceInput::report_insert_refused(owner)`, so the recording's R10.1 line says +// `outcome=refused`; a writable composer's line says `outcome=inserted`. +// +// The line is read through a process-wide logger. CAUTION for full-suite runs: this binary +// already installs one in `conversation_view/thread_view.rs` (`FileLinkWarningCapture`, which +// panics when another is set). The two never meet under the story's filters (`voice`, +// `voice_refused_log`), but a full `cargo test -p agent_ui` runs both in one process; whichever +// installs second fails. The executor must reconcile them (one shared capture, or nextest). +// +// Hostile halves, the wrong collapse first: a refused insert is not logged as inserted, and the +// refusal still shows its toast and leaves the composer's text as it was; then the wrong split: a +// writable composer's transcript is still logged as inserted; then the plain case. + +use std::sync::{Mutex, OnceLock}; +use std::thread::ThreadId; +use std::time::{Duration, Instant}; + +use gpui::TestAppContext; +use voice_input::state::{Input, Owner}; +use voice_input::{ViewState, VoiceInput}; + +use crate::voice_keys_tests::{REMOTE_SETTINGS, Thread, answering, init_voice, open_thread}; + +const TRANSCRIPT: &str = "Dictated private words"; + +struct Capture(Mutex>); + +impl log::Log for Capture { + fn enabled(&self, _: &log::Metadata) -> bool { + true + } + fn log(&self, record: &log::Record) { + self.0 + .lock() + .expect("lock") + .push((std::thread::current().id(), record.args().to_string())); + } + fn flush(&self) {} +} + +fn capture() -> &'static Capture { + static CAPTURE: OnceLock<&'static Capture> = OnceLock::new(); + CAPTURE.get_or_init(|| { + let capture: &'static Capture = Box::leak(Box::new(Capture(Mutex::new(Vec::new())))); + log::set_logger(capture).expect( + "another logger owns this test binary (thread_view.rs FileLinkWarningCapture): \ + run with `cargo test -p agent_ui voice_refused_log`", + ); + log::set_max_level(log::LevelFilter::Trace); + capture + }) +} + +fn recording_lines(capture: &Capture) -> Vec { + let me = std::thread::current().id(); + capture + .0 + .lock() + .expect("lock") + .iter() + .filter(|(thread, line)| *thread == me && line.starts_with("voice_input event=recording ")) + .map(|(_, line)| line.clone()) + .collect() +} + +fn outcome(line: &str) -> Option<&str> { + line.split(' ') + .find_map(|field| field.strip_prefix("outcome=")) +} + +fn text_of(thread: &Thread, cx: &mut TestAppContext) -> String { + let editor = thread.editor(cx); + cx.read(|cx| editor.read(cx).text(cx)) +} + +fn notifications(thread: &Thread, cx: &mut TestAppContext) -> usize { + thread + .workspace + .read_with(cx, |w, _| w.notification_ids().len()) +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +/// Starts a toggle recording in `thread`'s composer; `before_stop` runs while it records. +fn record( + thread: &Thread, + cx: &mut TestAppContext, + before_stop: impl FnOnce(&Thread, &mut TestAppContext), +) { + let id = thread.view(cx).entity_id(); + let owner = Owner(id.as_u64()); + let vi = cx.update(|cx| VoiceInput::global(cx)); + vi.update(cx, |vi, cx| vi.handle(Input::Click { owner }, cx)); + wait_until(cx, "recording", |cx| { + matches!( + vi.read_with(cx, |vi, _| vi.view_state(id)), + ViewState::Recording { .. } + ) + }); + before_stop(thread, cx); + vi.update(cx, |vi, cx| vi.handle(Input::Click { owner }, cx)); +} + +#[gpui::test] +async fn voice_refused_log_read_only_composer_logs_refused_and_shows_the_toast( + cx: &mut TestAppContext, +) { + let capture = capture(); + let answer: &'static str = Box::leak(format!(r#"{{"text": "{TRANSCRIPT}"}}"#).into_boxed_str()); + let _voice = init_voice(REMOTE_SETTINGS, answering(answer), cx); + let thread = open_thread(cx).await; + let toasts_before = notifications(&thread, cx); + + // Writable when the recording starts (a read-only composer starts nothing, R1.3), read-only + // by the time its transcript arrives. + record(&thread, cx, |thread, cx| { + let editor = thread.editor(cx); + editor.update(cx, |editor, cx| editor.set_read_only(true, cx)); + }); + wait_until(cx, "the refusal toast", |cx| { + notifications(&thread, cx) > toasts_before + }); + cx.run_until_parked(); + + assert_eq!( + text_of(&thread, cx), + "", + "R4.1 nothing inserted into a read-only composer" + ); + assert_eq!( + notifications(&thread, cx), + toasts_before + 1, + "the refusal is shown, once" + ); + let lines = recording_lines(capture); + assert_eq!( + lines.len(), + 1, + "R10.1 one line for the recording: {lines:?}" + ); + assert_eq!( + outcome(&lines[0]), + Some("refused"), + "R10.1 a refused insert is logged as refused, not inserted: {lines:?}" + ); + assert!( + !lines[0].contains("Dictated") && !lines[0].contains("private"), + "R10.3 no transcript in the line: {}", + lines[0] + ); +} + +#[gpui::test] +async fn voice_refused_log_writable_composer_logs_inserted(cx: &mut TestAppContext) { + let capture = capture(); + let answer: &'static str = Box::leak(format!(r#"{{"text": "{TRANSCRIPT}"}}"#).into_boxed_str()); + let _voice = init_voice(REMOTE_SETTINGS, answering(answer), cx); + let thread = open_thread(cx).await; + + record(&thread, cx, |_, _| {}); + wait_until(cx, "the transcript in the composer", |cx| { + !text_of(&thread, cx).is_empty() + }); + cx.run_until_parked(); + + let lines = recording_lines(capture); + assert_eq!(lines.len(), 1, "{lines:?}"); + assert_eq!(outcome(&lines[0]), Some("inserted"), "{lines:?}"); +} diff --git a/crates/reqwest_client/src/reqwest_client.rs b/crates/reqwest_client/src/reqwest_client.rs index 436ec60..356bbcf 100644 --- a/crates/reqwest_client/src/reqwest_client.rs +++ b/crates/reqwest_client/src/reqwest_client.rs @@ -1,4 +1,5 @@ use std::error::Error; +use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr}; use std::sync::{LazyLock, OnceLock}; use std::{borrow::Cow, mem, pin::Pin, task::Poll, time::Duration}; @@ -118,6 +119,35 @@ impl ReqwestClient { client.user_agent = Some(user_agent); Ok(client) } + + /// A client that never goes through a proxy: not Zed's, and not the ones reqwest would read + /// from the environment (`HTTP_PROXY`, `ALL_PROXY`, …) or the system. For traffic that must + /// stay on this machine, such as a plain-http request to a loopback service. It carries + /// `user_agent` and the TLS configuration of the app's client. + pub fn without_proxy(user_agent: &str) -> anyhow::Result { + let user_agent = HeaderValue::from_str(user_agent)?; + let client = Self::without_proxy_builder(&user_agent).build()?; + let mut client: ReqwestClient = client.into(); + client.user_agent = Some(user_agent); + Ok(client) + } + + /// The configuration of [`ReqwestClient::without_proxy`], before it is built. + fn without_proxy_builder(user_agent: &HeaderValue) -> reqwest::ClientBuilder { + let mut map = HeaderMap::new(); + map.insert(http::header::USER_AGENT, user_agent.clone()); + // `localhost` is answered here rather than by the system resolver, which on some hosts + // sends it to DNS with a search domain appended. Port 0 keeps the URL's own port. + let loopback = [ + SocketAddr::from((Ipv6Addr::LOCALHOST, 0)), + SocketAddr::from((Ipv4Addr::LOCALHOST, 0)), + ]; + Self::builder(None) + .no_proxy() + .resolve_to_addrs("localhost", &loopback) + .default_headers(map) + .use_preconfigured_tls(http_client_tls::tls_config()) + } } pub fn runtime() -> &'static tokio::runtime::Runtime { @@ -320,6 +350,11 @@ impl http_client::HttpClient for ReqwestClient { } } +#[cfg(test)] +mod reqwest_client_loopback_pin_tests; +#[cfg(test)] +mod reqwest_client_without_proxy_tests; + #[cfg(test)] mod tests { use std::io::{BufRead as _, BufReader, Read as _, Write as _}; diff --git a/crates/reqwest_client/src/reqwest_client_loopback_pin_tests.rs b/crates/reqwest_client/src/reqwest_client_loopback_pin_tests.rs new file mode 100644 index 0000000..3f911e9 --- /dev/null +++ b/crates/reqwest_client/src/reqwest_client_loopback_pin_tests.rs @@ -0,0 +1,69 @@ +// Story 044, task 8.4 fix round 1 (tech review L1) — the proxy-less client resolves `localhost` +// to the loopback addresses itself, so a plain-http request admitted as loopback never asks the +// system resolver, which on some hosts sends `localhost` to DNS with a search domain appended. +// +// The built `reqwest::Client` does not show its DNS overrides (its `Debug` lists only request +// settings), so the pin is read from the builder `without_proxy` builds from. That the URL's own +// port is still used (the override's port 0) is checked by a request to a real loopback server. + +use std::io::{BufRead as _, BufReader, Write as _}; +use std::net::TcpListener; +use std::sync::mpsc; +use std::time::Duration; + +use http_client::{AsyncBody, HttpClient, Request as HttpRequest}; +use reqwest::header::HeaderValue; + +use crate::ReqwestClient; + +const USER_AGENT: &str = "Zeo-voice-test/0.0"; + +#[test] +fn without_proxy_loopback_pin_maps_localhost_to_both_loopback_addresses() { + let builder = format!( + "{:?}", + ReqwestClient::without_proxy_builder(&HeaderValue::from_static(USER_AGENT)) + ); + assert!( + builder.contains(r#"dns_overrides: {"localhost": [[::1]:0, 127.0.0.1:0]}"#), + "localhost is pinned to ::1 and 127.0.0.1, never resolved by the system: {builder}" + ); +} + +#[test] +fn without_proxy_loopback_pin_keeps_the_port_the_url_names() { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind"); + let port = listener.local_addr().expect("address").port(); + let (request_line_tx, request_line_rx) = mpsc::channel::(); + let server = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().expect("accept"); + let mut reader = BufReader::new(stream.try_clone().expect("clone")); + let mut request_line = String::new(); + reader.read_line(&mut request_line).expect("read"); + loop { + let mut line = String::new(); + if reader.read_line(&mut line).expect("read") == 0 || line == "\r\n" { + break; + } + } + request_line_tx.send(request_line).ok(); + stream + .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n") + .expect("write"); + }); + + let client = ReqwestClient::without_proxy(USER_AGENT).expect("client"); + let request = HttpRequest::get(format!("http://localhost:{port}/v1/audio/transcriptions")) + .body(AsyncBody::default()) + .expect("request"); + let response = futures::executor::block_on(client.send(request)).expect("sent"); + assert!(response.status().is_success()); + let request_line = request_line_rx + .recv_timeout(Duration::from_secs(5)) + .expect("the server on 127.0.0.1 at the URL's port got the request"); + server.join().expect("server"); + assert!( + request_line.starts_with("GET /v1/audio/transcriptions"), + "{request_line:?}" + ); +} diff --git a/crates/reqwest_client/src/reqwest_client_without_proxy_tests.rs b/crates/reqwest_client/src/reqwest_client_without_proxy_tests.rs new file mode 100644 index 0000000..1399f65 --- /dev/null +++ b/crates/reqwest_client/src/reqwest_client_without_proxy_tests.rs @@ -0,0 +1,91 @@ +// Story 044, task 8.4 (refine v3) — `ReqwestClient::without_proxy(user_agent)`, the proxy-less client +// voice input sends plain-http loopback requests through (R8.9; design.md C1.10). Wired in +// `crates/reqwest_client/src/reqwest_client.rs` as +// +// #[cfg(test)] +// mod reqwest_client_without_proxy_tests; +// +// Validation filter: `without_proxy` (`cargo test -p reqwest_client without_proxy`). +// +// Contract: `ReqwestClient::without_proxy(user_agent: &str) -> anyhow::Result` +// reports no proxy, sends the given user agent like the app's client, and reaches a plain-http +// loopback server directly. +// +// Gap: that it ignores HTTP_PROXY/ALL_PROXY (reqwest's `no_proxy()`) is not exercised: setting +// them needs `set_var`, which is `unsafe` in edition 2024 and races every other test in the binary. +// +// Hostile half, the wrong collapse first: a proxy-less client is not a bare client — it still +// carries the app's user agent (the request a loopback service sees names Zeo); then the plain +// case: no proxy is reported, and the request arrives. + +use std::io::{BufRead as _, BufReader, Write as _}; +use std::net::TcpListener; +use std::sync::mpsc; +use std::time::Duration; + +use http_client::{AsyncBody, HttpClient, Request as HttpRequest}; + +use crate::ReqwestClient; + +const USER_AGENT: &str = "Zeo-voice-test/0.0"; + +#[test] +fn without_proxy_reports_no_proxy_and_keeps_the_user_agent() { + let client = ReqwestClient::without_proxy(USER_AGENT).expect("client"); + assert_eq!( + client.proxy(), + None, + "R8.9 the proxy-less client reports no proxy" + ); + assert_eq!( + client.user_agent().and_then(|value| value.to_str().ok()), + Some(USER_AGENT), + "the same user agent as the app's client" + ); +} + +#[test] +fn without_proxy_reaches_a_loopback_server_directly_with_its_user_agent() { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind"); + let address = listener.local_addr().expect("address"); + let (headers_tx, headers_rx) = mpsc::channel::>(); + let server = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().expect("accept"); + let mut reader = BufReader::new(stream.try_clone().expect("clone")); + let mut headers = Vec::new(); + loop { + let mut line = String::new(); + if reader.read_line(&mut line).expect("read") == 0 || line == "\r\n" { + break; + } + headers.push(line.trim_end().to_string()); + } + headers_tx.send(headers).ok(); + stream + .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n") + .expect("write"); + }); + + let client = ReqwestClient::without_proxy(USER_AGENT).expect("client"); + let request = HttpRequest::get(format!("http://{address}/v1/audio/transcriptions")) + .body(AsyncBody::default()) + .expect("request"); + let response = futures::executor::block_on(client.send(request)).expect("sent"); + assert!(response.status().is_success()); + let headers = headers_rx + .recv_timeout(Duration::from_secs(5)) + .expect("the loopback server got the request"); + server.join().expect("server"); + + assert!( + headers + .first() + .is_some_and(|line| line.starts_with("GET /v1/audio/transcriptions")), + "a direct request names the path, not an absolute proxy URL: {headers:?}" + ); + assert!( + headers.iter().any(|line| line.to_ascii_lowercase() + == format!("user-agent: {}", USER_AGENT.to_ascii_lowercase())), + "the request carries the app's user agent: {headers:?}" + ); +} diff --git a/crates/settings/src/vscode_import.rs b/crates/settings/src/vscode_import.rs index 0b5cacf..1c9775c 100644 --- a/crates/settings/src/vscode_import.rs +++ b/crates/settings/src/vscode_import.rs @@ -242,6 +242,7 @@ impl VsCodeSettings { title_bar: None, vim: None, vim_mode: None, + voice_input: None, workspace: self.workspace_settings_content(), which_key: None, modeline_lines: None, diff --git a/crates/settings_content/src/settings_content.rs b/crates/settings_content/src/settings_content.rs index 92a6c6b..2410607 100644 --- a/crates/settings_content/src/settings_content.rs +++ b/crates/settings_content/src/settings_content.rs @@ -324,6 +324,10 @@ pub struct SettingsContent { /// Settings related to Vim mode in Zed. pub vim: Option, + /// Settings for dictating into the agent composer: the microphone button, + /// the speech-to-text model and the optional remote service. + pub voice_input: Option, + /// Number of lines to search for modelines at the beginning and end of files. /// Modelines contain editor directives (e.g., vim/emacs settings) that configure /// the editor behavior for specific files. @@ -484,6 +488,61 @@ impl std::ops::DerefMut for FeatureFlagsMap { } } +/// Configuration for voice input in the agent composer. +#[with_fallible_options] +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize, JsonSchema, MergeFrom)] +pub struct VoiceInputSettingsContent { + /// Whether the composer shows the microphone button and binds the voice shortcut. + /// + /// Default: true + pub enabled: Option, + /// The local Whisper model transcribing the speech. + /// + /// Default: "base" + pub model: Option, + /// The spoken language: "auto" lets the model detect it, or a two-letter + /// lowercase code such as "en" or "pt". + /// + /// Default: "auto" + pub language: Option, + /// An OpenAI-compatible transcription service used instead of the local model. + pub remote: Option, +} + +/// The local Whisper models voice input can download and run. +#[derive( + Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize, JsonSchema, MergeFrom, +)] +#[serde(rename_all = "lowercase")] +pub enum VoiceModelContent { + Tiny, + #[default] + Base, + Small, +} + +/// Configuration for the remote transcription service of voice input. +#[with_fallible_options] +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize, JsonSchema, MergeFrom)] +pub struct VoiceInputRemoteSettingsContent { + /// Whether the audio is sent to the remote service instead of the local model. + /// + /// Default: false + pub enabled: Option, + /// The transcription endpoint (https, or http on loopback). `null` sets none. + /// + /// Default: null + pub url: Option, + /// The model name sent to the remote service. + /// + /// Default: "whisper-1" + pub model: Option, + /// The environment variable holding the remote service's API key. + /// + /// Default: "ZEO_VOICE_API_KEY" + pub api_key_env: Option, +} + impl SettingsContent { pub fn languages_mut(&mut self) -> &mut HashMap { &mut self.project.all_languages.languages.0 @@ -496,6 +555,7 @@ fallible_options::flattened_deserialize!(SettingsContent { call_hierarchy, command_palette, file_finder, git_panel, git_graph, tabs, tab_bar, tab_switcher, status_bar, preview_tabs, agent, agent_servers, audio, auto_update, base_keymap, collaboration_panel, debugger, diagnostics, git, + voice_input, global_lsp_settings, image_viewer, markdown_preview, repl, helix_mode, hide_mouse, journal, log, line_indicator_format, language_models, copilot, outline_panel, project_panel, node, proxy, reduce_motion, server_url, credentials_url, session, telemetry, terminal, diff --git a/crates/voice_input/Cargo.toml b/crates/voice_input/Cargo.toml new file mode 100644 index 0000000..f0c0bb9 --- /dev/null +++ b/crates/voice_input/Cargo.toml @@ -0,0 +1,41 @@ +[package] +name = "voice_input" +version = "0.1.0" +edition.workspace = true +publish = false +license = "GPL-3.0-or-later" +description = "Dictation into the agent composer: microphone capture and speech-to-text" + +[lints] +workspace = true + +[lib] +path = "src/voice_input.rs" +doctest = false + +[dependencies] +anyhow.workspace = true +audio.workspace = true +cpal.workspace = true +futures.workspace = true +gpui.workspace = true +http_client.workspace = true +log.workspace = true +paths.workspace = true +serde = { workspace = true, features = ["derive"] } +serde_json.workspace = true +settings.workspace = true +rodio.workspace = true +sha2.workspace = true +smol.workspace = true +tempfile.workspace = true +url.workspace = true +whisper-rs.workspace = true + +[target.'cfg(unix)'.dependencies] +libc.workspace = true + +[dev-dependencies] +gpui = { workspace = true, features = ["test-support"] } +http_client = { workspace = true, features = ["test-support"] } +settings = { workspace = true, features = ["test-support"] } diff --git a/crates/voice_input/LICENSE-GPL b/crates/voice_input/LICENSE-GPL new file mode 120000 index 0000000..89e542f --- /dev/null +++ b/crates/voice_input/LICENSE-GPL @@ -0,0 +1 @@ +../../LICENSE-GPL \ No newline at end of file diff --git a/crates/voice_input/src/audio_prep.rs b/crates/voice_input/src/audio_prep.rs new file mode 100644 index 0000000..ccabb24 --- /dev/null +++ b/crates/voice_input/src/audio_prep.rs @@ -0,0 +1,89 @@ +//! Audio preparation for speech-to-text: 16 kHz mono PCM, the silence gate and the WAV +//! encoding sent to a remote server (R4.6, R8.2). + +use rodio::buffer::SamplesBuffer; +use rodio::nz; +use rodio::source::UniformSourceIterator; +use std::num::NonZero; + +/// The sample rate whisper expects. +const WHISPER_RATE: NonZero = nz!(16_000); +/// Whisper takes one channel. +const MONO: NonZero = nz!(1); +/// 0.5 s at 16 kHz (R4.6). +const MIN_SAMPLES: usize = 8_000; +/// About -50 dBFS (R4.6). +const MIN_RMS: f32 = 0.003; +/// Size of the canonical RIFF/WAVE header with a 16-byte PCM `fmt ` chunk. +const WAV_HEADER_LEN: usize = 44; + +/// Resamples mono `samples` recorded at `rate` Hz to 16 kHz mono, clamped to [-1, 1]. +/// A `rate` of 0 or an empty input returns an empty vector. +pub fn to_whisper_pcm(samples: &[f32], rate: u32) -> Vec { + let Some(rate) = NonZero::new(rate) else { + return Vec::new(); + }; + if samples.is_empty() { + return Vec::new(); + } + let source = SamplesBuffer::new(MONO, rate, samples.to_vec()); + UniformSourceIterator::new(source, MONO, WHISPER_RATE) + .map(|sample| { + if sample.is_nan() { + 0.0 + } else { + sample.clamp(-1.0, 1.0) + } + }) + .collect() +} + +/// Root mean square level of `samples`; 0 for an empty slice. +pub fn rms(samples: &[f32]) -> f32 { + if samples.is_empty() { + return 0.0; + } + let sum_of_squares: f64 = samples + .iter() + .map(|&sample| f64::from(sample) * f64::from(sample)) + .sum(); + (sum_of_squares / samples.len() as f64).sqrt() as f32 +} + +/// True when 16 kHz `pcm16k` is shorter than 0.5 s or quieter than RMS 0.003 (R4.6). +pub fn speechless(pcm16k: &[f32]) -> bool { + pcm16k.len() < MIN_SAMPLES || rms(pcm16k) < MIN_RMS +} + +/// Encodes 16 kHz mono `pcm16k` as a 16-bit PCM WAV file (R8.2). +pub fn wav_pcm16(pcm16k: &[f32]) -> Vec { + const CHANNELS: u16 = MONO.get(); + const BITS_PER_SAMPLE: u16 = 16; + const BLOCK_ALIGN: u16 = CHANNELS * BITS_PER_SAMPLE / 8; + const BYTE_RATE: u32 = WHISPER_RATE.get() * BLOCK_ALIGN as u32; + + // A recording long enough to overflow these fields (37 hours) is far past the recording + // limit; saturate rather than panic. + let data_len = u32::try_from(pcm16k.len() * usize::from(BLOCK_ALIGN)).unwrap_or(u32::MAX); + let riff_len = data_len.saturating_add((WAV_HEADER_LEN - 8) as u32); + + let mut wav = Vec::with_capacity(WAV_HEADER_LEN + pcm16k.len() * usize::from(BLOCK_ALIGN)); + wav.extend_from_slice(b"RIFF"); + wav.extend_from_slice(&riff_len.to_le_bytes()); + wav.extend_from_slice(b"WAVE"); + wav.extend_from_slice(b"fmt "); + wav.extend_from_slice(&16u32.to_le_bytes()); + wav.extend_from_slice(&1u16.to_le_bytes()); + wav.extend_from_slice(&CHANNELS.to_le_bytes()); + wav.extend_from_slice(&WHISPER_RATE.get().to_le_bytes()); + wav.extend_from_slice(&BYTE_RATE.to_le_bytes()); + wav.extend_from_slice(&BLOCK_ALIGN.to_le_bytes()); + wav.extend_from_slice(&BITS_PER_SAMPLE.to_le_bytes()); + wav.extend_from_slice(b"data"); + wav.extend_from_slice(&data_len.to_le_bytes()); + for &sample in pcm16k { + let value = (sample.clamp(-1.0, 1.0) * f32::from(i16::MAX)).round() as i16; + wav.extend_from_slice(&value.to_le_bytes()); + } + wav +} diff --git a/crates/voice_input/src/audio_prep_tests.rs b/crates/voice_input/src/audio_prep_tests.rs new file mode 100644 index 0000000..8b272d1 --- /dev/null +++ b/crates/voice_input/src/audio_prep_tests.rs @@ -0,0 +1,139 @@ +// Story 044, task 2.2 — audio preparation (R4.6, R8.2). Wired in the crate root +// `src/voice_input.rs` as +// +// #[cfg(test)] +// mod audio_prep_tests; +// +// Validation filter: `audio_prep` (every test path is `audio_prep_tests::…`). +// +// Contract (design.md C1.5): `to_whisper_pcm(&[f32], u32) -> Vec` (16 kHz mono, clamped to +// [-1, 1]), `rms(&[f32]) -> f32`, `speechless(&[f32]) -> bool` (`len < 8_000 || rms < 0.003`), +// `wav_pcm16(&[f32]) -> Vec` (RIFF/WAVE, PCM, 1 channel, 16 000 Hz, 16-bit). +// +// Hostile halves: 7 999 samples are speechless AND 8 000 are not; RMS 0.0029 is speechless AND +// 0.0031 is not; 48 kHz input shrinks to a third AND 16 kHz input keeps its length (feeding 48 kHz +// to whisper gives fluent nonsense, not an error); out-of-range samples clamp AND in-range ones pass. + +use crate::audio_prep::{rms, speechless, to_whisper_pcm, wav_pcm16}; + +fn tone(len: usize, rate: u32, amplitude: f32) -> Vec { + (0..len) + .map(|i| amplitude * (2.0 * std::f32::consts::PI * 440.0 * i as f32 / rate as f32).sin()) + .collect() +} + +fn constant(len: usize, value: f32) -> Vec { + vec![value; len] +} + +#[test] +fn audio_prep_48k_resamples_to_a_third() { + let input = tone(48_000, 48_000, 0.5); // one second + let out = to_whisper_pcm(&input, 48_000); + let expected = 16_000i64; + assert!( + (out.len() as i64 - expected).abs() <= 16, + "len {} ≈ {expected}", + out.len() + ); +} + +#[test] +fn audio_prep_16k_keeps_its_length() { + let input = tone(16_000, 16_000, 0.5); + let out = to_whisper_pcm(&input, 16_000); + assert!((out.len() as i64 - 16_000).abs() <= 4, "len {}", out.len()); +} + +#[test] +fn audio_prep_44k1_resamples_proportionally() { + let out = to_whisper_pcm(&tone(44_100, 44_100, 0.5), 44_100); + assert!((out.len() as i64 - 16_000).abs() <= 16, "len {}", out.len()); +} + +#[test] +fn audio_prep_clamps_out_of_range_samples() { + let out = to_whisper_pcm(&constant(16_000, 3.0), 16_000); + assert!( + out.iter().all(|s| (-1.0..=1.0).contains(s)), + "clamped to [-1, 1]" + ); + let quiet = to_whisper_pcm(&constant(16_000, 0.25), 16_000); + let mid = quiet[quiet.len() / 2]; + assert!( + (mid - 0.25).abs() < 0.01, + "in-range samples pass through: {mid}" + ); +} + +#[test] +fn audio_prep_zero_rate_or_empty_input_is_empty() { + assert!(to_whisper_pcm(&[0.1, 0.2], 0).is_empty()); + assert!(to_whisper_pcm(&[], 48_000).is_empty()); +} + +#[test] +fn audio_prep_rms_of_known_signals() { + assert_eq!(rms(&[]), 0.0); + assert!((rms(&constant(100, 0.5)) - 0.5).abs() < 1e-6); + assert!( + (rms(&constant(100, -0.5)) - 0.5).abs() < 1e-6, + "sign does not matter" + ); + let sine = tone(16_000, 16_000, 1.0); + assert!((rms(&sine) - std::f32::consts::FRAC_1_SQRT_2).abs() < 0.01); +} + +#[test] +fn audio_prep_half_second_gate() { + assert!(speechless(&constant(7_999, 0.1)), "R4.6 under 0.5 s"); + assert!( + !speechless(&constant(8_000, 0.1)), + "exactly 0.5 s is enough" + ); +} + +#[test] +fn audio_prep_minus_50_dbfs_gate() { + assert!(speechless(&constant(16_000, 0.0029)), "R4.6 below 0.003"); + assert!(!speechless(&constant(16_000, 0.0031)), "above 0.003"); + assert!(speechless(&constant(16_000, 0.0)), "silence"); +} + +#[test] +fn audio_prep_wav_header_bytes() { + let pcm = [0.0f32, 1.0, -1.0, 0.5]; + let wav = wav_pcm16(&pcm); + assert_eq!(wav.len(), 44 + pcm.len() * 2); + let u32_at = |i: usize| u32::from_le_bytes(wav[i..i + 4].try_into().expect("4 bytes")); + let u16_at = |i: usize| u16::from_le_bytes(wav[i..i + 2].try_into().expect("2 bytes")); + assert_eq!(&wav[0..4], b"RIFF"); + assert_eq!(u32_at(4) as usize, wav.len() - 8, "RIFF chunk size"); + assert_eq!(&wav[8..12], b"WAVE"); + assert_eq!(&wav[12..16], b"fmt "); + assert_eq!(u32_at(16), 16, "PCM fmt chunk size"); + assert_eq!(u16_at(20), 1, "PCM"); + assert_eq!(u16_at(22), 1, "mono"); + assert_eq!(u32_at(24), 16_000, "16 kHz"); + assert_eq!(u32_at(28), 32_000, "byte rate"); + assert_eq!(u16_at(32), 2, "block align"); + assert_eq!(u16_at(34), 16, "16-bit"); + assert_eq!(&wav[36..40], b"data"); + assert_eq!(u32_at(40) as usize, pcm.len() * 2); + let sample = |n: usize| i16::from_le_bytes(wav[44 + 2 * n..46 + 2 * n].try_into().expect("2")); + assert_eq!(sample(0), 0); + assert!(sample(1) >= 32_766, "full scale positive: {}", sample(1)); + assert!(sample(2) <= -32_767, "full scale negative: {}", sample(2)); + assert!( + (sample(3) as i32 - 16_384).abs() <= 1, + "half scale: {}", + sample(3) + ); +} + +#[test] +fn audio_prep_wav_of_nothing_is_a_bare_header() { + let wav = wav_pcm16(&[]); + assert_eq!(wav.len(), 44); + assert_eq!(&wav[36..40], b"data"); +} diff --git a/crates/voice_input/src/capture.rs b/crates/voice_input/src/capture.rs new file mode 100644 index 0000000..4ad6af4 --- /dev/null +++ b/crates/voice_input/src/capture.rs @@ -0,0 +1,562 @@ +//! Microphone capture on a dedicated thread. +//! +//! cpal's `Stream` is `!Send`, so it is created, played and dropped on the `voice-capture` +//! thread; only samples (through a shared buffer) and [`CaptureEvent`]s leave that thread. +//! `audio::open_input_stream` is deliberately not used: rodio's `Microphone` iterator blocks +//! and has no error callback, so a stalled device would hang the reader and hide the failure. + +use std::{ + sync::{ + Arc, Mutex, PoisonError, + atomic::{AtomicBool, AtomicU8, Ordering}, + mpsc, + }, + thread::JoinHandle, + time::Duration, +}; + +use anyhow::{Context as _, Result, anyhow}; +use cpal::{ + DeviceId, ErrorKind, SampleFormat, SizedSample, StreamConfig, + traits::{DeviceTrait as _, StreamTrait as _}, +}; + +/// The longest recording kept, in seconds (R2.6). +const MAX_SECONDS: usize = 120; + +/// How long `start` waits for the device to open before giving up (R9.1). +const OPEN_TIMEOUT: Duration = Duration::from_secs(10); + +const THREAD_NAME: &str = "voice-capture"; + +/// Whether a `voice-capture` thread is alive. A thread whose open timed out is detached, not +/// joined, and may still hold cpal's process-wide ALSA lock: no second thread is started until it +/// has exited. +static CAPTURE_THREAD_LIVE: AtomicBool = AtomicBool::new(false); + +/// Clears [`CAPTURE_THREAD_LIVE`] when the capture thread exits, however it exits. +struct LiveGuard; + +impl Drop for LiveGuard { + fn drop(&mut self) { + CAPTURE_THREAD_LIVE.store(false, Ordering::SeqCst); + } +} + +/// Why a recording ended early; the entity maps these to state-machine inputs. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum CaptureEvent { + /// 120 s of audio are held; nothing more is appended (R2.6). + Limit, + /// The device stopped delivering audio; the reason is shown to the user (R9.2). + DeviceError(String), +} + +/// Why a recording ended. The entity learns of an early end through [`CaptureEvent`]; this is +/// what the recording itself remembers. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum Ended { + /// Stopped by the user (or by the entity on the user's behalf). + #[default] + User, + /// The 120 s limit was reached (R2.6). + Limit, + /// The device stopped delivering audio (R9.2). + DeviceError, +} + +impl Ended { + fn to_u8(self) -> u8 { + match self { + Self::User => 0, + Self::Limit => 1, + Self::DeviceError => 2, + } + } + + fn from_u8(value: u8) -> Self { + match value { + 1 => Self::Limit, + 2 => Self::DeviceError, + _ => Self::User, + } + } +} + +/// The first early end seen by the stream's callbacks, shared with the [`Recorder`]. +#[derive(Clone, Default)] +struct EndFlag(Arc); + +impl EndFlag { + /// Records `ended` unless an earlier end is already recorded. + fn set(&self, ended: Ended) { + // Losing the race means an earlier end is already recorded, which is the one kept. + let _ = self.0.compare_exchange( + Ended::User.to_u8(), + ended.to_u8(), + Ordering::SeqCst, + Ordering::SeqCst, + ); + } + + fn get(&self) -> Ended { + Ended::from_u8(self.0.load(Ordering::SeqCst)) + } +} + +/// What a finished recording hands back: mono samples at the device rate. +#[derive(Clone, Debug, Default, PartialEq)] +pub struct Recording { + pub samples: Vec, + pub sample_rate: u32, + pub ended: Ended, +} + +/// The channel a capture reports its events on. Named here so a fake backend in another crate +/// does not need its own `smol` dependency. +pub type CaptureEventSender = smol::channel::Sender; + +/// The seam the entity starts a recording through; tests inject a fake. +pub trait CaptureBackend: Send + Sync { + /// Opens `device` (or the system default input) and starts recording. Returns only once + /// the stream is playing, or with the reason it could not be opened (R9.1). + fn start( + &self, + device: Option, + events: CaptureEventSender, + ) -> Result>; +} + +/// A recording in progress. +pub trait ActiveCapture: Send { + /// Stops recording and returns what was captured. + fn stop(self: Box) -> Recording; + /// Stops recording and discards the audio. + fn cancel(self: Box); +} + +/// The production backend: cpal on a dedicated thread. +pub struct Cpal; + +impl CaptureBackend for Cpal { + fn start( + &self, + device: Option, + events: smol::channel::Sender, + ) -> Result> { + let requested = requested_label(device.as_ref()); + if CAPTURE_THREAD_LIVE + .compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst) + .is_err() + { + log::warn!("voice_input event=capture_refused reason=previous_thread_alive"); + return Err(anyhow!( + "a previous recording has not released the input device yet" + )); + } + let samples = Arc::new(Mutex::new(Vec::new())); + let ended = EndFlag::default(); + let (ready_tx, ready_rx) = mpsc::sync_channel::>(1); + let (stop_tx, stop_rx) = mpsc::channel::<()>(); + + let thread_samples = samples.clone(); + let thread_ended = ended.clone(); + let spawned = std::thread::Builder::new() + .name(THREAD_NAME.into()) + .spawn(move || { + let _live = LiveGuard; + capture_thread( + device, + events, + thread_samples, + thread_ended, + ready_tx, + stop_rx, + ) + }); + let thread = match spawned { + Ok(thread) => thread, + Err(error) => { + CAPTURE_THREAD_LIVE.store(false, Ordering::SeqCst); + return Err(error).context("could not start the capture thread"); + } + }; + + let mut recorder = Recorder { + stop: Some(stop_tx), + thread: Some(thread), + samples, + sample_rate: 0, + ended, + }; + match ready_rx.recv_timeout(OPEN_TIMEOUT) { + Ok(Ok(sample_rate)) => { + recorder.sample_rate = sample_rate; + Ok(Box::new(recorder)) + } + Ok(Err(err)) => { + log::warn!( + "voice_input event=capture_failed device={requested:?} reason={:?}", + format!("{err:#}") + ); + Err(err) + } + Err(mpsc::RecvTimeoutError::Timeout) => { + // The thread is left to finish on its own: dropping `recorder` would join it, + // and joining a thread stuck inside the device would hang the caller instead. + recorder.thread = None; + log::warn!("voice_input event=capture_open_timeout device={requested:?}"); + Err(anyhow!( + "{requested} did not open within {} s", + OPEN_TIMEOUT.as_secs() + )) + } + Err(mpsc::RecvTimeoutError::Disconnected) => { + log::warn!("voice_input event=capture_thread_died device={requested:?}"); + Err(anyhow!( + "the capture thread for {requested} stopped unexpectedly" + )) + } + } + } +} + +/// Handle to the `voice-capture` thread; dropping it stops the thread. +pub struct Recorder { + /// Dropping this sender is the stop signal: the thread drops its stream and exits. + stop: Option>, + thread: Option>, + samples: Arc>>, + sample_rate: u32, + ended: EndFlag, +} + +impl Recorder { + /// Stops the thread and waits for it, so the stream's callback can no longer append. + fn shutdown(&mut self) { + drop(self.stop.take()); + if let Some(thread) = self.thread.take() + && thread.join().is_err() + { + log::warn!("voice_input event=capture_thread_panicked"); + } + } + + fn take_samples(&self) -> Vec { + std::mem::take(&mut *self.samples.lock().unwrap_or_else(PoisonError::into_inner)) + } +} + +impl ActiveCapture for Recorder { + fn stop(mut self: Box) -> Recording { + self.shutdown(); + let samples = self.take_samples(); + log::info!( + "voice_input event=capture_stopped samples={} rate={}", + samples.len(), + self.sample_rate + ); + Recording { + samples, + sample_rate: self.sample_rate, + ended: self.ended.get(), + } + } + + fn cancel(mut self: Box) { + self.shutdown(); + drop(self.take_samples()); + log::info!("voice_input event=capture_cancelled"); + } +} + +impl Drop for Recorder { + fn drop(&mut self) { + self.shutdown(); + } +} + +/// The body of the `voice-capture` thread: open, report, wait for stop, drop the stream. +fn capture_thread( + device: Option, + events: smol::channel::Sender, + samples: Arc>>, + ended: EndFlag, + ready: mpsc::SyncSender>, + stop: mpsc::Receiver<()>, +) { + let (stream, sample_rate) = match open_stream(device.as_ref(), events, samples, ended) { + Ok(opened) => opened, + Err(err) => { + if ready.send(Err(err)).is_err() { + log::debug!("voice_input event=capture_open_unreported"); + } + return; + } + }; + if ready.send(Ok(sample_rate)).is_err() { + // `start` gave up waiting; nobody owns this recording. + return; + } + match stop.recv() { + // The recorder never sends: dropping its sender is the stop signal, so a disconnect + // is the expected way out, not a failure. + Ok(()) | Err(mpsc::RecvError) => drop(stream), + } +} + +/// Resolves the device and builds a playing input stream for its default config. +/// Runs on the capture thread only: the returned stream is `!Send`. +fn open_stream( + device: Option<&DeviceId>, + events: smol::channel::Sender, + samples: Arc>>, + ended: EndFlag, +) -> Result<(cpal::Stream, u32)> { + let requested = requested_label(device); + let device = audio::resolve_device(device, true) + .with_context(|| format!("could not open {requested}"))?; + let name = device_label(&device); + let supported = device + .default_input_config() + .with_context(|| format!("could not read the input format of {name:?}"))?; + let format = supported.sample_format(); + check_format(format).with_context(|| format!("{name:?} cannot be recorded"))?; + + let sample_rate = supported.sample_rate(); + let channels = supported.channels(); + let limit = limit_samples(sample_rate); + samples + .lock() + .unwrap_or_else(PoisonError::into_inner) + .reserve_exact(limit); + let sink = Sink { + samples, + channels, + limit, + events: events.clone(), + ended: ended.clone(), + stopped: false, + // Reserved here so the first real-time callback does not allocate. + scratch: Vec::with_capacity(16_384), + }; + + let config = supported.config(); + let stream = match format { + SampleFormat::F32 => build::(&device, config, sink, events, ended, |s| s), + SampleFormat::I16 => build::(&device, config, sink, events, ended, i16_to_f32), + SampleFormat::U16 => build::(&device, config, sink, events, ended, u16_to_f32), + SampleFormat::I24 => build::(&device, config, sink, events, ended, i24_to_f32), + SampleFormat::I32 => build::(&device, config, sink, events, ended, i32_to_f32), + other => return Err(unsupported_format(other)), + } + .with_context(|| format!("could not open an input stream on {name:?}"))?; + stream + .play() + .with_context(|| format!("could not start recording from {name:?}"))?; + + log::info!( + "voice_input event=capture_started device={name:?} rate={sample_rate} channels={channels} format={:?}", + format.to_string() + ); + Ok((stream, sample_rate)) +} + +fn build( + device: &cpal::Device, + config: StreamConfig, + mut sink: Sink, + events: smol::channel::Sender, + ended: EndFlag, + convert: fn(T) -> f32, +) -> std::result::Result { + let mut reported = false; + device.build_input_stream::( + config, + move |data: &[T], _info| sink.push(data, convert), + move |err: cpal::Error| { + if !is_fatal(err.kind()) { + log::debug!( + "voice_input event=capture_glitch reason={:?}", + err.to_string() + ); + return; + } + if !reported { + reported = true; + ended.set(Ended::DeviceError); + send(&events, CaptureEvent::DeviceError(err.to_string())); + } + }, + // A finite poll timeout keeps cpal's ALSA worker re-checking for a device that vanished + // without a hang-up event; `None` would wait forever and the recording would go silent. + Some(Duration::from_millis(500)), + ) +} + +/// The data callback's state: converts, downmixes and appends, then signals `Limit` once. +struct Sink { + samples: Arc>>, + channels: u16, + limit: usize, + events: smol::channel::Sender, + ended: EndFlag, + /// Set once the limit is reached or the buffer is unusable; later callbacks do nothing. + stopped: bool, + /// Reused across callbacks so the audio thread allocates only while it grows. + scratch: Vec, +} + +impl Sink { + fn push(&mut self, data: &[T], convert: fn(T) -> f32) { + if self.stopped { + return; + } + self.scratch.clear(); + self.scratch.extend(data.iter().copied().map(convert)); + let reached = match self.samples.lock() { + Ok(mut buf) => append_mono(&mut buf, &self.scratch, self.channels, self.limit), + Err(_) => { + self.stopped = true; + self.ended.set(Ended::DeviceError); + send( + &self.events, + CaptureEvent::DeviceError("the recording buffer was poisoned".into()), + ); + return; + } + }; + if reached { + self.stopped = true; + self.ended.set(Ended::Limit); + send(&self.events, CaptureEvent::Limit); + } + } +} + +fn send(events: &smol::channel::Sender, event: CaptureEvent) { + if events.try_send(event).is_err() { + log::debug!("voice_input event=capture_event_dropped"); + } +} + +/// Whether a stream error means the device stopped delivering audio (R9.2). An xrun, a +/// reroute to a new default device or a refused real-time promotion leave the stream running. +fn is_fatal(kind: ErrorKind) -> bool { + !matches!( + kind, + ErrorKind::Xrun | ErrorKind::DeviceChanged | ErrorKind::RealtimeDenied + ) +} + +fn requested_label(device: Option<&DeviceId>) -> String { + match device { + Some(id) => format!("the input device {:?}", id.to_string()), + None => "the default input device".to_string(), + } +} + +fn device_label(device: &cpal::Device) -> String { + match device.description() { + Ok(description) => description.name().to_string(), + Err(_) => match device.id() { + Ok(id) => id.to_string(), + Err(_) => "unknown input device".to_string(), + }, + } +} + +fn unsupported_format(format: SampleFormat) -> anyhow::Error { + anyhow!("unsupported sample format {format}") +} + +/// The number of mono samples held at most: 120 s at `rate` (R2.6). +pub fn limit_samples(rate: u32) -> usize { + (rate as usize).saturating_mul(MAX_SECONDS) +} + +/// Averages each interleaved frame of `channels` samples into one mono sample appended to +/// `buf`, never growing `buf` past `limit`. Returns `true` only on the call that makes +/// `buf.len()` reach `limit`. A trailing partial frame is ignored. +pub fn append_mono(buf: &mut Vec, interleaved: &[f32], channels: u16, limit: usize) -> bool { + if channels == 0 || buf.len() >= limit { + return false; + } + let room = limit - buf.len(); + let divisor = f32::from(channels); + buf.extend( + interleaved + .chunks_exact(usize::from(channels)) + .take(room) + .map(|frame| frame.iter().sum::() / divisor), + ); + buf.len() >= limit +} + +/// Maps a signed 16-bit sample to [-1, 1). +pub fn i16_to_f32(s: i16) -> f32 { + f32::from(s) / 32_768.0 +} + +/// Maps an unsigned 16-bit sample (origin 32 768) to [-1, 1). +pub fn u16_to_f32(s: u16) -> f32 { + (f32::from(s) - 32_768.0) / 32_768.0 +} + +/// Full scale of a signed 24-bit sample: 2^23. +const I24_SCALE: f64 = 8_388_608.0; + +/// Full scale of a signed 32-bit sample: 2^31. +const I32_SCALE: f64 = 2_147_483_648.0; + +/// Maps a signed 24-bit sample (cpal's `I24`: 24 significant bits in an `i32`) to [-1, 1). +/// +/// cpal passes S24_LE words through as read, and the top byte is not promised to be a sign +/// extension (PulseAudio's S24_32 writer zero-fills it), so the sign is taken from bit 23: the +/// left shift drops the top byte and the arithmetic right shift extends bit 23 over it. The +/// result is always in -2^23..=2^23-1, so no clamp is needed. +pub fn i24_to_f32(s: cpal::I24) -> f32 { + let value = (s.inner() << 8) >> 8; + (f64::from(value) / I24_SCALE) as f32 +} + +/// Maps a signed 32-bit sample to [-1, 1]. +pub fn i32_to_f32(s: i32) -> f32 { + (f64::from(s) / I32_SCALE).clamp(-1.0, 1.0) as f32 +} + +/// `F32`, `I16`, `U16`, `I24` and `I32` are recorded; any other format is refused by name. +pub fn check_format(format: SampleFormat) -> Result<()> { + match format { + SampleFormat::F32 + | SampleFormat::I16 + | SampleFormat::U16 + | SampleFormat::I24 + | SampleFormat::I32 => Ok(()), + other => Err(unsupported_format(other)), + } +} + +#[cfg(test)] +mod fatal_tests { + use super::*; + + #[test] + fn capture_glitches_do_not_stop_the_recording() { + for kind in [ + ErrorKind::Xrun, + ErrorKind::DeviceChanged, + ErrorKind::RealtimeDenied, + ] { + assert!(!is_fatal(kind), "{kind:?}"); + } + for kind in [ + ErrorKind::DeviceNotAvailable, + ErrorKind::StreamInvalidated, + ErrorKind::BackendError, + ErrorKind::Other, + ] { + assert!(is_fatal(kind), "{kind:?}"); + } + } +} diff --git a/crates/voice_input/src/capture_formats_tests.rs b/crates/voice_input/src/capture_formats_tests.rs new file mode 100644 index 0000000..145f1b7 --- /dev/null +++ b/crates/voice_input/src/capture_formats_tests.rs @@ -0,0 +1,141 @@ +// Story 044, task 8.6 (refine v3) — I24 and I32 microphones are recorded (R9.1, MODIFIED in v3; +// design.md C1.4). Wired in the crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod capture_formats_tests; +// +// Validation filter: `capture` (every test path is `capture_formats_tests::…`). +// +// Contract: `capture::check_format(cpal::SampleFormat) -> anyhow::Result<()>` accepts F32, I16, +// U16, I24 and I32 and refuses anything else naming the format; `capture::i24_to_f32(cpal::I24) +// -> f32` divides by 2^23 and `capture::i32_to_f32(i32) -> f32` by 2^31, both within [-1, 1]. +// +// Hostile halves, the wrong collapse first: U24 and U32 sit next to I24 and I32 (same width, +// unsigned) and stay refused, as do I8, I64, F64, U8 and the DSD containers; then the accepted +// five. For the scale, a value each converter would map wrongly if it borrowed the other's +// divisor: 2^22 in a 24-bit sample is half scale, and 2^23 in a 32-bit sample is 1/256 — a +// 24-bit divisor applied to I32 would clip it to full scale, a 31-bit divisor applied to I24 +// would leave it near silence. + +use cpal::{I24, SampleFormat}; + +use crate::capture::{check_format, i24_to_f32, i32_to_f32}; + +const EPSILON: f32 = 1e-6; + +fn i24(value: i32) -> I24 { + I24::new(value).expect("a value inside the 24-bit range") +} + +#[test] +fn capture_formats_unsigned_and_other_widths_are_refused_by_name() { + for bad in [ + SampleFormat::U24, + SampleFormat::U32, + SampleFormat::I8, + SampleFormat::I64, + SampleFormat::F64, + SampleFormat::U8, + SampleFormat::U64, + SampleFormat::DsdU8, + SampleFormat::DsdU32, + ] { + let error = check_format(bad).expect_err("refused"); + let shown = format!("{error:#}").to_lowercase(); + let name = format!("{bad:?}").to_lowercase(); + assert!( + shown.contains(&name), + "R9.1 the error names {name}: {shown}" + ); + } +} + +#[test] +fn capture_formats_accepts_f32_i16_u16_i24_and_i32() { + for ok in [ + SampleFormat::F32, + SampleFormat::I16, + SampleFormat::U16, + SampleFormat::I24, + SampleFormat::I32, + ] { + assert!( + check_format(ok).is_ok(), + "R9.1 {ok:?} is recorded: {:?}", + check_format(ok) + ); + } +} + +#[test] +fn capture_formats_i32_spans_the_unit_range() { + assert_eq!( + i32_to_f32(i32::MIN), + -1.0, + "i32::MIN is full negative scale" + ); + assert!( + (i32_to_f32(i32::MAX) - 1.0).abs() < EPSILON, + "i32::MAX ≈ 1: {}", + i32_to_f32(i32::MAX) + ); + assert_eq!(i32_to_f32(0), 0.0, "0 is silence"); +} + +#[test] +fn capture_formats_i24_spans_the_unit_range() { + let min = -(1 << 23); + let max = (1 << 23) - 1; + assert_eq!(i24_to_f32(i24(min)), -1.0, "-2^23 is full negative scale"); + assert!( + (i24_to_f32(i24(max)) - 1.0).abs() < 1e-6, + "2^23-1 ≈ 1: {}", + i24_to_f32(i24(max)) + ); + assert_eq!(i24_to_f32(i24(0)), 0.0, "0 is silence"); +} + +#[test] +fn capture_formats_each_width_uses_its_own_divisor() { + assert!( + (i24_to_f32(i24(1 << 22)) - 0.5).abs() < EPSILON, + "2^22 in a 24-bit sample is half scale, not near silence: {}", + i24_to_f32(i24(1 << 22)) + ); + assert!( + (i32_to_f32(1 << 30) - 0.5).abs() < EPSILON, + "2^30 in a 32-bit sample is half scale: {}", + i32_to_f32(1 << 30) + ); + assert!( + (i32_to_f32(1 << 23) - 1.0 / 256.0).abs() < EPSILON, + "2^23 in a 32-bit sample is 1/256, not clipped to full scale: {}", + i32_to_f32(1 << 23) + ); +} + +#[test] +fn capture_formats_keep_the_sign() { + for value in [-1, -12_345, -(1 << 20)] { + assert!(i24_to_f32(i24(value)) < 0.0, "I24 {value} stays negative"); + assert!(i32_to_f32(value) < 0.0, "I32 {value} stays negative"); + assert!( + i24_to_f32(i24(-value)) > 0.0, + "I24 {} stays positive", + -value + ); + assert!(i32_to_f32(-value) > 0.0, "I32 {} stays positive", -value); + } +} + +#[test] +fn capture_formats_stay_within_the_unit_range() { + for value in [i32::MIN, i32::MIN + 1, -1, 0, 1, i32::MAX - 1, i32::MAX] { + let sample = i32_to_f32(value); + assert!((-1.0..=1.0).contains(&sample), "I32 {value} → {sample}"); + } + for value in [-(1 << 23), -(1 << 23) + 1, 0, (1 << 23) - 2, (1 << 23) - 1] { + let sample = i24_to_f32(i24(value)); + assert!((-1.0..=1.0).contains(&sample), "I24 {value} → {sample}"); + } +} diff --git a/crates/voice_input/src/capture_i24_sign_tests.rs b/crates/voice_input/src/capture_i24_sign_tests.rs new file mode 100644 index 0000000..9ff5c08 --- /dev/null +++ b/crates/voice_input/src/capture_i24_sign_tests.rs @@ -0,0 +1,61 @@ +// Story 044, task 8.6, fix round 1 — an I24 sample is sign-extended from bit 23 (R9.1). +// +// cpal hands S24_LE over as raw 32-bit words, and the top byte is not promised to be a sign +// extension: PulseAudio's S24_32 writer zero-fills it. Read as-is, every negative sample would +// land in [2^23, 2^24) and turn into +1.0, so the converter must look at the low 24 bits only. + +use cpal::I24; + +use crate::capture::i24_to_f32; + +const EPSILON: f32 = 1e-6; + +fn raw(word: i32) -> I24 { + I24::new_unchecked(word) +} + +#[test] +fn capture_i24_zero_filled_minus_one_is_negative() { + let sample = i24_to_f32(raw(0x00FF_FFFF)); + let expected = -1.0 / 8_388_608.0; + assert!(sample < 0.0, "0x00FF_FFFF is -1 in 24 bits: {sample}"); + assert!( + (sample - expected).abs() < EPSILON, + "0x00FF_FFFF ≈ -1/2^23: {sample}" + ); +} + +#[test] +fn capture_i24_zero_filled_minimum_is_full_negative_scale() { + assert_eq!( + i24_to_f32(raw(0x0080_0000)), + -1.0, + "0x0080_0000 is -2^23 in 24 bits" + ); +} + +#[test] +fn capture_i24_maximum_is_just_under_full_scale() { + let sample = i24_to_f32(raw(0x007F_FFFF)); + assert!( + sample > 0.0 && sample <= 1.0, + "0x007F_FFFF in (0, 1]: {sample}" + ); + assert!((sample - 1.0).abs() < EPSILON, "0x007F_FFFF ≈ 1: {sample}"); +} + +#[test] +fn capture_i24_ignores_the_top_byte() { + for (garbage, clean) in [ + (0x12FF_FFFF_u32 as i32, 0x00FF_FFFF), + (0xFFFF_FFFF_u32 as i32, 0x00FF_FFFF), + (0x5580_0000_u32 as i32, 0x0080_0000), + (0xAB00_1234_u32 as i32, 0x0000_1234), + ] { + assert_eq!( + i24_to_f32(raw(garbage)), + i24_to_f32(raw(clean)), + "top byte of {garbage:#010x} is ignored" + ); + } +} diff --git a/crates/voice_input/src/capture_tests.rs b/crates/voice_input/src/capture_tests.rs new file mode 100644 index 0000000..737cff3 --- /dev/null +++ b/crates/voice_input/src/capture_tests.rs @@ -0,0 +1,106 @@ +// Story 044, task 3.2 — the pure helpers of the capture thread (R2.6, R9.1, R9.2). Wired in the +// crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod capture_tests; +// +// Validation filter: `capture` (every test path is `capture_tests::…`). +// +// No device is opened here: a unit test must never touch the operator's microphone. The device +// path (`start`, the `!Send` stream) is exercised by the GUI proof (7.3) and, through a fake +// backend, by `entity_tests`. +// +// Contract: design.md C1.4 (downmix by averaging, a 120 s cap that signals `Limit` once, `F32`/ +// `I16`/`U16` accepted and any other format refused by name). Assumed helper signatures — the +// cpal callback is a thin shell over them; adjust the `use` only: +// - `capture::limit_samples(rate: u32) -> usize` — 120 s × rate (mono) +// - `capture::append_mono(buf: &mut Vec, interleaved: &[f32], channels: usize, +// limit: usize) -> bool` — averages each frame into `buf`, never past +// `limit`; returns `true` only on the call that makes `buf.len()` reach `limit` +// - `capture::i16_to_f32(i16) -> f32`, `capture::u16_to_f32(u16) -> f32` +// - `capture::check_format(cpal::SampleFormat) -> anyhow::Result<()>` +// +// Hostile halves: a stereo frame of +1/-1 averages to 0 (not the left channel); the limit is +// signalled once AND later callbacks neither append nor signal again; F32/I16/U16 are accepted AND +// I32/F64/U8 are refused with their name. + +use crate::capture::{append_mono, check_format, i16_to_f32, limit_samples, u16_to_f32}; + +#[test] +fn capture_limit_is_120_seconds_of_mono() { + assert_eq!(limit_samples(48_000), 5_760_000); + assert_eq!(limit_samples(44_100), 5_292_000); + assert_eq!(limit_samples(16_000), 1_920_000); +} + +#[test] +fn capture_downmix_averages_each_frame() { + let mut buf = Vec::new(); + let reached = append_mono(&mut buf, &[1.0, -1.0, 0.5, 0.25, 0.0, 1.0], 2, 1_000); + assert!(!reached); + assert_eq!(buf, vec![0.0, 0.375, 0.5]); +} + +#[test] +fn capture_mono_passes_through_and_three_channels_average() { + let mut buf = Vec::new(); + append_mono(&mut buf, &[0.1, -0.2, 0.3], 1, 1_000); + assert_eq!(buf, vec![0.1, -0.2, 0.3]); + let mut buf = Vec::new(); + append_mono(&mut buf, &[0.3, 0.3, 0.3, 0.9, 0.0, 0.0], 3, 1_000); + assert!( + (buf[0] - 0.3).abs() < 1e-6 && (buf[1] - 0.3).abs() < 1e-6, + "{buf:?}" + ); +} + +#[test] +fn capture_limit_signals_once_and_stops_appending() { + let mut buf = Vec::new(); + assert!(!append_mono(&mut buf, &[0.1; 6], 1, 8)); + assert!( + append_mono(&mut buf, &[0.2; 6], 1, 8), + "the call that reaches the limit signals" + ); + assert_eq!(buf.len(), 8, "truncated exactly at the limit"); + assert!(!append_mono(&mut buf, &[0.3; 6], 1, 8), "no second Limit"); + assert_eq!(buf.len(), 8, "nothing appended after the limit"); +} + +#[test] +fn capture_limit_reached_exactly_by_a_whole_chunk() { + let mut buf = Vec::new(); + assert!(append_mono(&mut buf, &[0.1; 4], 1, 4)); + assert_eq!(buf.len(), 4); +} + +#[test] +fn capture_zero_channels_appends_nothing() { + let mut buf = Vec::new(); + assert!(!append_mono(&mut buf, &[0.1, 0.2], 0, 10)); + assert!(buf.is_empty()); +} + +#[test] +fn capture_integer_samples_convert_to_unit_range() { + assert_eq!(i16_to_f32(0), 0.0); + assert!((i16_to_f32(i16::MIN) + 1.0).abs() < 1e-4); + assert!((i16_to_f32(i16::MAX) - 1.0).abs() < 1e-4); + assert!((u16_to_f32(32_768)).abs() < 1e-4, "u16 midpoint is silence"); + assert!((u16_to_f32(0) + 1.0).abs() < 1e-4); + assert!((u16_to_f32(u16::MAX) - 1.0).abs() < 1e-4); +} + +#[test] +fn capture_accepts_f32_i16_u16_and_names_any_other_format() { + use cpal::SampleFormat; + for ok in [SampleFormat::F32, SampleFormat::I16, SampleFormat::U16] { + assert!(check_format(ok).is_ok(), "{ok:?}"); + } + for bad in [SampleFormat::F64, SampleFormat::U8] { + let err = check_format(bad).expect_err("refused"); + let shown = format!("{err:#}").to_lowercase(); + let name = format!("{bad:?}").to_lowercase(); + assert!(shown.contains(&name), "the error names {name}: {shown}"); + } +} diff --git a/crates/voice_input/src/deliver_tests.rs b/crates/voice_input/src/deliver_tests.rs new file mode 100644 index 0000000..31cc34e --- /dev/null +++ b/crates/voice_input/src/deliver_tests.rs @@ -0,0 +1,271 @@ +// Story 044, task 5.4 fix round 1 — a finished transcript is delivered to its composer only when +// that composer is still open, and the decision survives the gap between the transcript arriving +// and its delivery (R4.7; tech review 5.4 L1, L4). Wired in the crate root `src/voice_input.rs`. +// +// Both tests run the decisive steps inside one `cx.update`, so gpui flushes effects (and releases +// dropped entities) only when that update ends — the same-flush interleaving the review describes. + +use std::sync::{ + Arc, Mutex, + atomic::{AtomicUsize, Ordering}, +}; +use std::time::{Duration, Instant}; + +use futures::channel::oneshot; +use gpui::{AnyWeakEntity, AppContext as _, Entity, EntityId, TestAppContext, UpdateGlobal as _}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use settings::SettingsStore; + +use crate::capture::{ActiveCapture, CaptureBackend, CaptureEvent, Ended, Recording}; +use crate::state::{Input, Owner}; +use crate::{ViewState, VoiceEvent, VoiceInput, init_in}; + +const REMOTE: &str = r#"{ "voice_input": { "remote": { "enabled": true, + "url": "http://127.0.0.1:9/v1/audio/transcriptions", "api_key_env": "ZEO_VOICE_TEST_UNSET_044_D1" } } }"#; + +#[derive(Default)] +struct Mic { + starts: AtomicUsize, + stops: AtomicUsize, +} + +struct FakeBackend(Arc); +struct FakeActive(Arc); + +impl CaptureBackend for FakeBackend { + fn start( + &self, + _device: Option, + _events: smol::channel::Sender, + ) -> anyhow::Result> { + self.0.starts.fetch_add(1, Ordering::SeqCst); + Ok(Box::new(FakeActive(self.0.clone()))) + } +} + +impl ActiveCapture for FakeActive { + fn stop(self: Box) -> Recording { + self.0.stops.fetch_add(1, Ordering::SeqCst); + let rate = 48_000; + let samples = (0..rate).map(|i| 0.2 * (i as f32 * 0.05).sin()).collect(); + Recording { + samples, + sample_rate: rate, + ended: Ended::User, + } + } + fn cancel(self: Box) {} +} + +struct Harness { + vi: Entity, + events: Arc>>, + /// Answers the held remote request; the tests deliver their own `Transcribed` instead. + release: Option>, + _tmp: tempfile::TempDir, +} + +impl Harness { + fn delivered(&self, owner: EntityId) -> (usize, usize) { + let events = self.events.lock().expect("lock"); + let inserts = events + .iter() + .filter(|e| matches!(e, VoiceEvent::Insert { owner: o, .. } if *o == owner)) + .count(); + let discards = events + .iter() + .filter(|e| matches!(e, VoiceEvent::Discarded { owner: o } if *o == owner)) + .count(); + (inserts, discards) + } +} + +impl Drop for Harness { + fn drop(&mut self) { + if let Some(release) = self.release.take() { + release.send(()).ok(); + } + } +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +fn setup(cx: &mut TestAppContext) -> Harness { + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(REMOTE, cx) + .result() + .expect("settings parse"); + }) + }); + let (release, held) = oneshot::channel::<()>(); + let held = Arc::new(Mutex::new(Some(held))); + let http: Arc = FakeHttpClient::create(move |_req| { + let held = held.lock().expect("lock").take(); + async move { + if let Some(held) = held { + held.await.ok(); + } + Ok(Response::builder() + .status(200) + .body(AsyncBody::from(r#"{"text": "late"}"#))?) + } + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let backend: Arc = Arc::new(FakeBackend(Arc::default())); + cx.update(|cx| init_in(http, tmp.path().join("models"), backend, cx)); + let vi = cx.update(|cx| VoiceInput::global(cx)); + let events: Arc>> = Arc::default(); + let sink = events.clone(); + cx.update(|cx| { + cx.subscribe(&vi, move |_, event: &VoiceEvent, _| { + sink.lock().expect("lock").push(event.clone()) + }) + .detach() + }); + Harness { + vi, + events, + release: Some(release), + _tmp: tmp, + } +} + +/// Records for `owner` and stops, leaving the remote request held: the phase is `Transcribing`. +fn transcribing(h: &Harness, owner: EntityId, alive: AnyWeakEntity, cx: &mut TestAppContext) { + let o = Owner(owner.as_u64()); + h.vi.update(cx, |vi, _| vi.register_owner(o, alive)); + h.vi.update(cx, |vi, cx| vi.handle(Input::Click { owner: o }, cx)); + wait_until(cx, "recording", |cx| { + matches!( + h.vi.read_with(cx, |vi, _| vi.view_state(owner)), + ViewState::Recording { .. } + ) + }); + h.vi.update(cx, |vi, cx| vi.handle(Input::Click { owner: o }, cx)); + wait_until(cx, "transcribing", |cx| { + matches!( + h.vi.read_with(cx, |vi, _| vi.view_state(owner)), + ViewState::Transcribing + ) + }); +} + +/// L1: the transcript arrives (the machine is `Idle` again), the composer closes, and a new +/// composer registers — which prunes closed owners while `Idle` — all before the delivery runs. +#[gpui::test] +async fn deliver_closed_owner_pruned_before_delivery_is_discarded(cx: &mut TestAppContext) { + let h = setup(cx); + let a = cx.new(|_| ()); + let b = cx.new(|_| ()); + let ida = a.entity_id(); + transcribing(&h, ida, a.downgrade().into(), cx); + + let vi = h.vi.clone(); + let b_alive: AnyWeakEntity = b.downgrade().into(); + let idb = Owner(b.entity_id().as_u64()); + cx.update(move |cx| { + vi.update(cx, |vi, cx| { + vi.handle( + Input::Transcribed { + text: "for A".into(), + }, + cx, + ) + }); + drop(a); + vi.update(cx, |vi, _| vi.register_owner(idb, b_alive)); + }); + cx.run_until_parked(); + + assert_eq!( + h.delivered(ida), + (0, 1), + "R4.7: discarded, not inserted into a closed composer" + ); +} + +struct Parent { + _thread_view: Entity<()>, +} + +/// L4: the composer is held by a parent whose last handle is dropped in the same update the +/// transcript arrives in. The composer itself still upgrades until the flush releases the parent, +/// so only a delivery decided after that flush sees it closed. +#[gpui::test] +async fn deliver_owner_released_with_its_parent_in_the_same_update_is_discarded( + cx: &mut TestAppContext, +) { + let h = setup(cx); + let child = cx.new(|_| ()); + let id = child.entity_id(); + let alive: AnyWeakEntity = child.downgrade().into(); + let parent = cx.new(|_| Parent { + _thread_view: child, + }); + transcribing(&h, id, alive.clone(), cx); + + let vi = h.vi.clone(); + cx.update(move |cx| { + drop(parent); + assert!( + alive.upgrade().is_some(), + "the child outlives its parent until the flush" + ); + vi.update(cx, |vi, cx| { + vi.handle( + Input::Transcribed { + text: "for the child".into(), + }, + cx, + ) + }); + }); + cx.run_until_parked(); + + assert_eq!( + h.delivered(id), + (0, 1), + "R4.7: discarded once its parent is released" + ); +} + +/// The hostile half: an owner still open receives its transcript, once. +#[gpui::test] +async fn deliver_open_owner_receives_the_transcript(cx: &mut TestAppContext) { + let h = setup(cx); + let a = cx.new(|_| ()); + let ida = a.entity_id(); + transcribing(&h, ida, a.downgrade().into(), cx); + h.vi.update(cx, |vi, cx| { + vi.handle( + Input::Transcribed { + text: "for A".into(), + }, + cx, + ) + }); + cx.run_until_parked(); + assert_eq!(h.delivered(ida), (1, 0)); + drop(a); +} diff --git a/crates/voice_input/src/download.rs b/crates/voice_input/src/download.rs new file mode 100644 index 0000000..0ea4130 --- /dev/null +++ b/crates/voice_input/src/download.rs @@ -0,0 +1,442 @@ +//! Model download: fetch one model from its pinned URL, verify it, move it into place. + +use std::fmt; +use std::fs; +use std::io::Write as _; +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use anyhow::Context as _; +use futures::channel::oneshot; +use futures::{AsyncReadExt as _, FutureExt as _}; +use http_client::{AsyncBody, HttpClient}; +use sha2::{Digest as _, Sha256}; + +use crate::models::{ModelSpec, VoiceModel, url}; + +const CHUNK: usize = 64 * 1024; +const PROGRESS_INTERVAL: Duration = Duration::from_millis(250); +/// How long the server may take to send the response headers, or the body may stay connected +/// without delivering a byte, before the download fails. +pub const STALL_TIMEOUT: Duration = Duration::from_secs(30); +/// Every staging file's name starts with this; [`sweep_staging`] matches nothing else. +const STAGING_PREFIX: &str = ".download-"; + +#[derive(Debug)] +pub enum DownloadError { + Cancelled, + Http(anyhow::Error), + Io(anyhow::Error), + Mismatch { + model: VoiceModel, + expected: &'static str, + got: String, + bytes: u64, + }, +} + +impl DownloadError { + fn outcome(&self) -> &'static str { + match self { + Self::Cancelled => "cancelled", + Self::Http(_) => "http", + Self::Io(_) => "io", + Self::Mismatch { .. } => "mismatch", + } + } +} + +impl fmt::Display for DownloadError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Cancelled => write!(f, "the voice model download was cancelled"), + Self::Http(error) => write!(f, "could not download the voice model: {error:#}"), + Self::Io(error) => write!(f, "could not save the voice model: {error:#}"), + Self::Mismatch { + model, + expected, + got, + bytes, + } => write!( + f, + "the downloaded {} voice model did not match its checksum and was deleted \ + (expected SHA-256 {expected}, got {got} over {bytes} bytes)", + model_name(*model) + ), + } + } +} + +impl std::error::Error for DownloadError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Http(error) | Self::Io(error) => Some(error.as_ref()), + Self::Cancelled | Self::Mismatch { .. } => None, + } + } +} + +pub(crate) fn model_name(model: VoiceModel) -> &'static str { + match model { + VoiceModel::Tiny => "tiny", + VoiceModel::Base => "base", + VoiceModel::Small => "small", + } +} + +/// Logs exactly one `model_download` line when the download ends — including when the future is +/// dropped mid-download, which is reported as `cancelled` (R10.2). +struct DownloadLog { + model: VoiceModel, + started: Instant, + bytes: u64, + outcome: &'static str, +} + +impl Drop for DownloadLog { + fn drop(&mut self) { + let level = match self.outcome { + "ok" | "cancelled" => log::Level::Info, + _ => log::Level::Warn, + }; + log::log!( + level, + "voice_input event=model_download model={} bytes={} duration_ms={} outcome={}", + model_name(self.model), + self.bytes, + self.started.elapsed().as_millis(), + self.outcome + ); + } +} + +/// Downloads `spec` from its pinned URL into `dir/`, verifying its size and SHA-256 first. +/// +/// The bytes are staged in `dir/.download--` — the same directory as the final +/// path, so the closing `rename` never crosses filesystems. The staging file is removed on every +/// path but success, including this future being dropped mid-download (R6.5, R6.6). +/// +/// `progress` receives the bytes received so far, at most every 250 ms, never past `spec.size`. +/// Sending on `cancel` stops the download with [`DownloadError::Cancelled`]; dropping its sender +/// only means it can no longer be cancelled that way. A server that sends no response headers, or +/// a body that stays connected but delivers no byte, for [`STALL_TIMEOUT`] fails with +/// [`DownloadError::Http`] (R6.13). +/// +/// The file I/O is blocking `std::fs` — a write per 64 KiB chunk and an `fsync` of up to 488 MB — +/// so this future must run on a background executor (`cx.background_spawn`), never on the +/// foreground thread. +pub async fn download_model( + http: Arc, + spec: &'static ModelSpec, + dir: PathBuf, + progress: impl Fn(u64) + Send + 'static, + cancel: oneshot::Receiver<()>, +) -> Result { + download_model_with_stall(http, spec, dir, progress, cancel, STALL_TIMEOUT).await +} + +/// [`download_model`] with the stall timeout as a parameter, so a test need not wait 30 s. +#[doc(hidden)] +pub async fn download_model_with_stall( + http: Arc, + spec: &'static ModelSpec, + dir: PathBuf, + progress: impl Fn(u64) + Send + 'static, + cancel: oneshot::Receiver<()>, + stall: Duration, +) -> Result { + let mut log = DownloadLog { + model: spec.model, + started: Instant::now(), + bytes: 0, + outcome: "cancelled", + }; + let result = download( + http.as_ref(), + spec, + &dir, + progress, + cancel, + stall, + &mut log.bytes, + ) + .await; + log.outcome = match &result { + Ok(_) => "ok", + Err(error) => error.outcome(), + }; + result +} + +async fn download( + http: &dyn HttpClient, + spec: &'static ModelSpec, + dir: &Path, + progress: impl Fn(u64), + cancel: oneshot::Receiver<()>, + stall: Duration, + received: &mut u64, +) -> Result { + fs::create_dir_all(dir) + .with_context(|| format!("creating the models directory {dir:?}")) + .map_err(DownloadError::Io)?; + + let url = url(spec); + let mut cancel = cancel.fuse(); + // Hugging Face answers `resolve/` URLs with a redirect to its CDN: follow it. + let mut request = http.get(&url, AsyncBody::empty(), true).fuse(); + // A server that accepts the connection but never answers would otherwise hold the download + // forever: the app's client sets no read timeout (R6.13). + #[allow(clippy::disallowed_methods)] + let mut no_response = smol::Timer::after(stall).fuse(); + let answered = loop { + futures::select_biased! { + cancelled = cancel => match cancelled { + Ok(()) => return Err(DownloadError::Cancelled), + Err(oneshot::Canceled) => continue, + }, + answered = request => break answered, + _ = no_response => { + return Err(DownloadError::Http(anyhow::anyhow!( + "no response from {url} within {} s", + stall.as_secs_f64() + ))); + } + } + }; + let mut response = answered + .with_context(|| format!("requesting {url}")) + .map_err(DownloadError::Http)?; + let status = response.status(); + if !status.is_success() { + return Err(DownloadError::Http(anyhow::anyhow!( + "{url} answered {status}" + ))); + } + + // `NamedTempFile` is the drop guard: it removes the staging file whenever it is dropped + // without being persisted, whichever way this function returns or stops being polled. + let mut staging = tempfile::Builder::new() + .prefix(&format!("{STAGING_PREFIX}{}-", spec.file)) + .tempfile_in(dir) + .with_context(|| format!("creating a staging file in {dir:?}")) + .map_err(DownloadError::Io)?; + // Held until the rename (dropping the file releases it): it tells another Zeo's + // `sweep_staging` this file is live, which its age cannot (R6.12). The file was just created + // under a random name, so a failure means the filesystem has no locks: the age rule is then + // the only guard, and the download goes on. + if let Err(error) = staging.as_file().try_lock() { + log::warn!( + "voice_input event=staging_lock_failed path={:?} error={:?}", + staging.path().display().to_string(), + error.to_string() + ); + } + + let mut hasher = Sha256::new(); + let mut buffer = vec![0; CHUNK]; + let mut last_progress: Option = None; + let body = response.body_mut(); + loop { + // The app's client sets no read timeout, and HTTP/2 keep-alive only catches a dead peer, + // not a live one that stops sending. `smol::Timer` because this future runs outside gpui + // and takes no executor. + #[allow(clippy::disallowed_methods)] + let mut stalled = smol::Timer::after(stall).fuse(); + let read = futures::select_biased! { + cancelled = cancel => match cancelled { + Ok(()) => return Err(DownloadError::Cancelled), + // The sender is gone: nobody can cancel any more, so keep reading. + Err(oneshot::Canceled) => continue, + }, + read = body.read(&mut buffer).fuse() => read, + _ = stalled => { + return Err(DownloadError::Http(anyhow::anyhow!( + "no data received for {} s from {url}", + stall.as_secs_f64() + ))); + } + }; + let n = read + .with_context(|| format!("reading the body of {url}")) + .map_err(DownloadError::Http)?; + if n == 0 { + break; + } + *received += n as u64; + hasher.update(&buffer[..n]); + // The pinned size, never `Content-Length`: the header can be absent after the redirect, + // and a body longer than the pinned size cannot be the pinned file. + if *received > spec.size { + return Err(mismatch(spec, hasher, *received)); + } + staging + .as_file_mut() + .write_all(&buffer[..n]) + .with_context(|| format!("writing {:?}", staging.path())) + .map_err(DownloadError::Io)?; + if last_progress.is_none_or(|at| at.elapsed() >= PROGRESS_INTERVAL) { + progress(*received); + last_progress = Some(Instant::now()); + } + } + + let got = hex(hasher.clone()); + if *received != spec.size || got != spec.sha256 { + return Err(mismatch(spec, hasher, *received)); + } + + staging + .as_file() + .sync_all() + .with_context(|| format!("syncing {:?}", staging.path())) + .map_err(DownloadError::Io)?; + let destination = dir.join(spec.file); + staging + .persist(&destination) + .map_err(|error| { + // Dropping `error.file` removes the staging file. + anyhow::Error::new(error.error).context(format!( + "renaming {:?} to {destination:?}", + error.file.path() + )) + }) + .map_err(DownloadError::Io)?; + sync_directory(dir); + Ok(destination) +} + +/// Deletes the staging files a crashed or killed download left in `dir`, and returns how many. +/// +/// A file goes only when all three hold: it is a regular file named `.download-*` (a symlink or a +/// directory with that name is not a staging file, and a symlink is never followed); its last +/// write is older than `older_than`; and its exclusive lock can be taken. A running download holds +/// that lock from the staging file's creation until its rename, so a live file is kept however old +/// its mtime looks — the final fsync does not move it, and a suspend moves the wall clock without +/// the writer's stall timer (R6.12). On a filesystem without locks, the age rule decides alone: +/// in the body loop a live download writes at least every [`STALL_TIMEOUT`], but not during its +/// final fsync and rename. A file that cannot be read, locked or deleted is logged and skipped. +/// Blocking `std::fs`: run it on a background executor. +pub fn sweep_staging(dir: &Path, older_than: Duration) -> usize { + let entries = match fs::read_dir(dir) { + Ok(entries) => entries, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return 0, + Err(error) => { + warn_sweep(dir, &error); + return 0; + } + }; + let mut removed = 0; + for entry in entries { + let entry = match entry { + Ok(entry) => entry, + Err(error) => { + warn_sweep(dir, &error); + continue; + } + }; + if !entry + .file_name() + .to_str() + .is_some_and(|name| name.starts_with(STAGING_PREFIX)) + { + continue; + } + let path = entry.path(); + let stale = fs::symlink_metadata(&path).and_then(|metadata| { + if !metadata.file_type().is_file() { + return Ok(false); + } + let modified = metadata.modified()?; + // A modification time in the future reads as fresh: it is kept. + Ok(modified.elapsed().is_ok_and(|age| age > older_than)) + }); + match stale { + Ok(false) => {} + Ok(true) if !unlocked(&path) => {} + Ok(true) => match fs::remove_file(&path) { + Ok(()) => removed += 1, + // Another Zeo swept it first. + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => warn_sweep(&path, &error), + }, + Err(error) => warn_sweep(&path, &error), + } + } + if removed > 0 { + log::info!("voice_input event=staging_swept count={removed}"); + } + removed +} + +/// Whether no running download holds `path`'s lock. The probe lock is released before this +/// returns, so the caller can delete the file on every platform. +fn unlocked(path: &Path) -> bool { + let mut options = fs::OpenOptions::new(); + options.read(true).write(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + // Swapped for a symlink since its metadata was read: fail rather than follow it. + options.custom_flags(libc::O_NOFOLLOW); + } + let file = match options.open(path) { + Ok(file) => file, + // Another Zeo swept it first. + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return false, + Err(error) => { + warn_sweep(path, &error); + return false; + } + }; + match file.try_lock() { + Ok(()) => true, + Err(fs::TryLockError::WouldBlock) => false, + // No locks here, so no writer holds one either: the age rule decides. + Err(fs::TryLockError::Error(error)) if error.kind() == std::io::ErrorKind::Unsupported => { + true + } + Err(fs::TryLockError::Error(error)) => { + warn_sweep(path, &error); + false + } + } +} + +fn warn_sweep(path: &Path, error: &std::io::Error) { + log::warn!( + "voice_input event=staging_sweep_failed path={:?} error={:?}", + path.display().to_string(), + error.to_string() + ); +} + +/// Makes the rename durable. A failure is logged, not returned: the verified model is already in +/// place, and if a power loss undoes the rename, `model_on_disk` reports it missing and the user +/// is asked to download it again. +fn sync_directory(dir: &Path) { + #[cfg(unix)] + if let Err(error) = fs::File::open(dir).and_then(|directory| directory.sync_all()) { + log::warn!( + "voice_input event=models_dir_sync_failed path={:?} error={:?}", + dir.display().to_string(), + error.to_string() + ); + } + #[cfg(not(unix))] + let _ = dir; +} + +fn mismatch(spec: &'static ModelSpec, hasher: Sha256, bytes: u64) -> DownloadError { + DownloadError::Mismatch { + model: spec.model, + expected: spec.sha256, + got: hex(hasher), + bytes, + } +} + +fn hex(hasher: Sha256) -> String { + format!("{:x}", hasher.finalize()) +} diff --git a/crates/voice_input/src/download_owner_tests.rs b/crates/voice_input/src/download_owner_tests.rs new file mode 100644 index 0000000..96dc2b1 --- /dev/null +++ b/crates/voice_input/src/download_owner_tests.rs @@ -0,0 +1,589 @@ +// Story 044, task 8.5 (refine v3) — a model download outlives the composer that asked for it +// (R5.1, R6.3, R6.6, R6.14, R6.15; design.md C1.9 `AdoptDownload`, C1.10 abandonment). Wired in +// the crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod download_owner_tests; +// +// Validation filter: `download_owner` (every test path is `download_owner_tests::…`). +// +// Contract: `state::Input::AdoptDownload { owner }` — in `Downloading` the owner becomes `owner` +// and the effect is `Render`; in every other phase nothing happens. The entity keeps a running +// download when its composer closes: the next composer's start input adopts it (it then sees the +// progress and the Cancel control, and is not told busy) and is consumed; `DownloadCancel` still +// cancels it, and so does turning the feature off. A download that ends after its owner closed +// notifies nobody. +// +// The download is served by a body the test feeds by hand, so "still running" is observable: the +// body reader is never dropped, it keeps being read, and the staging file keeps growing. +// +// Gap: "it completes, with the model at the final path" cannot be driven — a model is only kept +// when its bytes hash to the pinned SHA-256, which a test cannot produce for a real model. The +// test instead lets the download run to the end of its body (a short body, so it ends as a +// checksum mismatch): the body was read to EOF and never dropped early, the staging file is +// deleted (R6.6) and the phase is Idle. +// +// Hostile halves, the wrong collapse first: with A still open, B is refused as busy and A keeps +// its download (adoption is not a free-for-all); once B adopted it, a third composer C is busy too, +// and B's own second click changes nothing. Then the wrong split: B, a different composer from the +// one that consented, does take the download over when A is gone. Then the plain cases. + +use std::{ + collections::VecDeque, + path::Path, + pin::Pin, + sync::{ + Arc, Mutex, + atomic::{AtomicUsize, Ordering}, + }, + task::{Context, Poll, Waker}, + time::{Duration, Instant}, +}; + +use futures::AsyncRead; +use gpui::{AppContext as _, Entity, EntityId, TestAppContext, UpdateGlobal as _}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use settings::SettingsStore; + +use crate::capture::{ActiveCapture, CaptureBackend, CaptureEvent}; +use crate::state::{Ctx, Effect, Input, Machine, Owner, Phase}; +use crate::{ViewState, VoiceEvent, VoiceInput, init_in}; + +// ── a body fed by the test ───────────────────────────────────────────────── + +#[derive(Default)] +struct Feed { + queue: VecDeque, + ended: bool, + waker: Option, + read: u64, + eof_returned: bool, + dropped: bool, +} + +#[derive(Clone, Default)] +struct Body(Arc>); + +impl Body { + fn push(&self, bytes: usize) { + let mut feed = self.0.lock().expect("lock"); + feed.queue.extend(std::iter::repeat_n(0x5a, bytes)); + if let Some(waker) = feed.waker.take() { + waker.wake(); + } + } + fn end(&self) { + let mut feed = self.0.lock().expect("lock"); + feed.ended = true; + if let Some(waker) = feed.waker.take() { + waker.wake(); + } + } + fn read(&self) -> u64 { + self.0.lock().expect("lock").read + } + fn dropped(&self) -> bool { + self.0.lock().expect("lock").dropped + } + fn eof_returned(&self) -> bool { + self.0.lock().expect("lock").eof_returned + } +} + +struct Reader(Body); + +impl AsyncRead for Reader { + fn poll_read( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut [u8], + ) -> Poll> { + let mut feed = (self.0).0.lock().expect("lock"); + if !feed.queue.is_empty() { + let n = buf.len().min(feed.queue.len()); + for (slot, byte) in buf.iter_mut().zip(feed.queue.drain(..n)) { + *slot = byte; + } + feed.read += n as u64; + return Poll::Ready(Ok(n)); + } + if feed.ended { + feed.eof_returned = true; + return Poll::Ready(Ok(0)); + } + feed.waker = Some(cx.waker().clone()); + Poll::Pending + } +} + +impl Drop for Reader { + fn drop(&mut self) { + if let Ok(mut feed) = (self.0).0.lock() { + feed.dropped = true; + } + } +} + +// ── harness ──────────────────────────────────────────────────────────────── + +struct NoMicrophone; + +impl CaptureBackend for NoMicrophone { + fn start( + &self, + _device: Option, + _events: smol::channel::Sender, + ) -> anyhow::Result> { + anyhow::bail!("no microphone: these tests never record") + } +} + +struct Harness { + vi: Entity, + body: Body, + requests: Arc, + events: Arc>>, + models: std::path::PathBuf, + _tmp: tempfile::TempDir, +} + +impl Harness { + fn send(&self, input: Input, cx: &mut TestAppContext) { + self.vi.update(cx, |vi, cx| vi.handle(input, cx)); + cx.run_until_parked(); + } + fn view(&self, owner: EntityId, cx: &mut TestAppContext) -> ViewState { + self.vi.read_with(cx, |vi, _| vi.view_state(owner)) + } + fn notices(&self) -> Vec<(EntityId, String)> { + self.events + .lock() + .expect("lock") + .iter() + .filter_map(|event| match event { + VoiceEvent::Notify { owner, message } => Some((*owner, message.clone())), + _ => None, + }) + .collect() + } + fn notices_for(&self, owner: EntityId) -> Vec { + self.notices() + .into_iter() + .filter(|(o, _)| *o == owner) + .map(|(_, message)| message) + .collect() + } + fn register(&self, owner: &Entity<()>, cx: &mut TestAppContext) { + let id = owner.entity_id(); + let weak = owner.downgrade().into(); + self.vi + .update(cx, |vi, _| vi.register_owner(Owner(id.as_u64()), weak)); + } + fn staging_len(&self) -> Option { + staging_files(&self.models) + .first() + .and_then(|path| std::fs::metadata(path).ok()) + .map(|metadata| metadata.len()) + } + /// Feeds `bytes` and waits until the download has read them. + fn feed(&self, bytes: usize, cx: &mut TestAppContext) { + let target = self.body.read() + bytes as u64; + self.body.push(bytes); + wait_until(cx, "the download to read the fed bytes", |_| { + self.body.read() >= target + }); + } +} + +fn staging_files(dir: &Path) -> Vec { + std::fs::read_dir(dir) + .map(|read| { + read.filter_map(|entry| entry.ok()) + .map(|entry| entry.path()) + .filter(|path| { + path.file_name() + .is_some_and(|name| name.to_string_lossy().starts_with(".download-")) + }) + .collect() + }) + .unwrap_or_default() +} + +fn set_settings(json: &str, cx: &mut TestAppContext) { + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(json, cx) + .result() + .expect("settings parse"); + }) + }); +} + +/// Local mode, the tiny model missing: a click asks for consent, and the consent downloads from a +/// body the test feeds. +fn setup(cx: &mut TestAppContext) -> Harness { + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); + set_settings(r#"{ "voice_input": { "model": "tiny" } }"#, cx); + let tmp = tempfile::tempdir().expect("tempdir"); + let models = tmp.path().join("voice").join("models"); + let body = Body::default(); + let requests = Arc::new(AtomicUsize::new(0)); + let http: Arc = { + let body = body.clone(); + let requests = requests.clone(); + FakeHttpClient::create(move |_| { + requests.fetch_add(1, Ordering::SeqCst); + let reader = Reader(body.clone()); + async move { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from_reader(reader))?) + } + }) + }; + let capture: Arc = Arc::new(NoMicrophone); + cx.update(|cx| init_in(http, models.clone(), capture, cx)); + let vi = cx.update(|cx| VoiceInput::global(cx)); + let events: Arc>> = Arc::default(); + let sink = events.clone(); + cx.update(|cx| { + cx.subscribe(&vi, move |_, event: &VoiceEvent, _| { + sink.lock().expect("lock").push(event.clone()) + }) + .detach() + }); + cx.run_until_parked(); + Harness { + vi, + body, + requests, + events, + models, + _tmp: tmp, + } +} + +fn o(id: EntityId) -> Owner { + Owner(id.as_u64()) +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +/// A consents and its download has read its first 64 KiB. +fn a_downloading(h: &Harness, cx: &mut TestAppContext) -> Entity<()> { + let a = cx.new(|_| ()); + h.register(&a, cx); + h.send( + Input::Click { + owner: o(a.entity_id()), + }, + cx, + ); + assert!( + matches!(h.view(a.entity_id(), cx), ViewState::AwaitingConsent { .. }), + "precondition: A is asked for consent: {:?}", + h.view(a.entity_id(), cx) + ); + h.send(Input::ConsentGiven, cx); + wait_until(cx, "the download request", |_| { + h.requests.load(Ordering::SeqCst) == 1 + }); + h.feed(64 * 1024, cx); + assert!( + matches!(h.view(a.entity_id(), cx), ViewState::Downloading { .. }), + "precondition: A's download runs: {:?}", + h.view(a.entity_id(), cx) + ); + a +} + +// ── the wrong collapse: adoption only when the owner is gone ─────────────── + +#[gpui::test] +async fn download_owner_another_composer_is_busy_while_the_owner_is_open(cx: &mut TestAppContext) { + let h = setup(cx); + let a = a_downloading(&h, cx); + let b = cx.new(|_| ()); + h.register(&b, cx); + + h.send( + Input::Click { + owner: o(b.entity_id()), + }, + cx, + ); + + assert_eq!( + h.notices_for(b.entity_id()), + vec!["Voice input is busy in another thread".to_string()], + "R5.1 B is refused while A, still open, downloads" + ); + assert!(matches!(h.view(b.entity_id(), cx), ViewState::Busy)); + assert!( + matches!(h.view(a.entity_id(), cx), ViewState::Downloading { .. }), + "A keeps its download: {:?}", + h.view(a.entity_id(), cx) + ); + assert!(!h.body.dropped(), "the download is not cancelled"); +} + +#[gpui::test] +async fn download_owner_a_third_composer_is_busy_after_the_adoption(cx: &mut TestAppContext) { + let h = setup(cx); + let a = a_downloading(&h, cx); + let b = cx.new(|_| ()); + let c = cx.new(|_| ()); + h.register(&b, cx); + h.register(&c, cx); + drop(a); + cx.run_until_parked(); + h.send( + Input::Click { + owner: o(b.entity_id()), + }, + cx, + ); + assert!( + matches!(h.view(b.entity_id(), cx), ViewState::Downloading { .. }), + "R6.15 B shows the download it adopted: {:?}", + h.view(b.entity_id(), cx) + ); + + h.send( + Input::Click { + owner: o(c.entity_id()), + }, + cx, + ); + assert_eq!( + h.notices_for(c.entity_id()), + vec!["Voice input is busy in another thread".to_string()], + "R5.1 C is refused: B, open, now owns the download" + ); + h.send( + Input::Click { + owner: o(b.entity_id()), + }, + cx, + ); + assert!( + h.notices_for(b.entity_id()).is_empty(), + "B's own click while its download runs changes nothing: {:?}", + h.notices_for(b.entity_id()) + ); + assert!(matches!( + h.view(b.entity_id(), cx), + ViewState::Downloading { .. } + )); + assert!(!h.body.dropped(), "still the one download"); + assert_eq!(h.requests.load(Ordering::SeqCst), 1); +} + +// ── the wrong split: the next composer takes the download over ───────────── + +#[gpui::test] +async fn download_owner_the_next_composer_adopts_a_download_whose_owner_closed( + cx: &mut TestAppContext, +) { + let h = setup(cx); + let a = a_downloading(&h, cx); + let staged = h.staging_len().expect("a staging file while downloading"); + drop(a); + cx.run_until_parked(); + + let b = cx.new(|_| ()); + h.register(&b, cx); + h.send( + Input::Click { + owner: o(b.entity_id()), + }, + cx, + ); + + assert!( + matches!(h.view(b.entity_id(), cx), ViewState::Downloading { .. }), + "R6.15 B sees the running download's progress and Cancel: {:?}", + h.view(b.entity_id(), cx) + ); + assert!( + h.notices_for(b.entity_id()).is_empty(), + "B is not told busy: {:?}", + h.notices_for(b.entity_id()) + ); + assert!(!h.body.dropped(), "R6.14 the download was not cancelled"); + assert_eq!( + h.requests.load(Ordering::SeqCst), + 1, + "one request, never restarted" + ); + + h.feed(64 * 1024, cx); + let grown = h.staging_len().expect("the staging file is still there"); + assert!( + grown > staged, + "the staging file keeps growing: {staged} → {grown}" + ); + + // R6.3: B's Cancel cancels the download it adopted. + h.send(Input::DownloadCancel, cx); + wait_until(cx, "the download to stop", |_| h.body.dropped()); + wait_until(cx, "the staging file to go", |_| { + staging_files(&h.models).is_empty() + }); + assert!( + matches!(h.view(b.entity_id(), cx), ViewState::Idle { .. }), + "R6.6 idle after the cancel: {:?}", + h.view(b.entity_id(), cx) + ); +} + +#[gpui::test] +async fn download_owner_a_download_whose_owner_closed_runs_to_its_end_and_notifies_nobody( + cx: &mut TestAppContext, +) { + let h = setup(cx); + let a = a_downloading(&h, cx); + drop(a); + cx.run_until_parked(); + + h.feed(256 * 1024, cx); + assert!( + !h.body.dropped(), + "R6.14 the download keeps running without its composer" + ); + // A body shorter than the pinned model: the download ends on its own, as a mismatch. + h.body.end(); + wait_until(cx, "the download to reach its end", |_| { + h.body.eof_returned() + }); + wait_until(cx, "the staging file to go", |_| { + staging_files(&h.models).is_empty() + }); + let other = cx.new(|_| ()).entity_id(); + wait_until(cx, "idle", |cx| { + matches!(h.view(other, cx), ViewState::Idle { .. }) + }); + assert_eq!(h.requests.load(Ordering::SeqCst), 1); + assert!( + h.notices().is_empty(), + "the end of a download whose composer closed notifies nobody (A is gone): {:?}", + h.notices() + ); +} + +// ── the plain cases ──────────────────────────────────────────────────────── + +#[gpui::test] +async fn download_owner_disabling_the_feature_cancels_a_running_download(cx: &mut TestAppContext) { + let h = setup(cx); + let a = a_downloading(&h, cx); + set_settings(r#"{ "voice_input": { "enabled": false } }"#, cx); + wait_until(cx, "the download to stop", |_| h.body.dropped()); + wait_until(cx, "the staging file to go", |_| { + staging_files(&h.models).is_empty() + }); + assert!(matches!(h.view(a.entity_id(), cx), ViewState::Hidden)); +} + +// ── the machine ──────────────────────────────────────────────────────────── + +const CTX: Ctx = Ctx { + enabled: true, + remote: false, + model_present: false, + composer_read_only: false, +}; + +/// A machine in `Downloading { owner: 1, bytes: 4096 }`, through the inputs that lead there. +fn downloading() -> Machine { + let mut machine = Machine::default(); + let now = Instant::now(); + machine.on(Input::Click { owner: Owner(1) }, now, &CTX); + machine.on(Input::ConsentGiven, now, &CTX); + machine.on(Input::DownloadProgress(4096), now, &CTX); + assert_eq!( + machine.phase(), + &Phase::Downloading { + owner: Owner(1), + bytes: 4096 + }, + "precondition" + ); + machine +} + +#[test] +fn download_owner_adopt_download_changes_the_owner_and_keeps_the_progress() { + let mut machine = downloading(); + let effects = machine.on( + Input::AdoptDownload { owner: Owner(2) }, + Instant::now(), + &CTX, + ); + assert_eq!( + machine.phase(), + &Phase::Downloading { + owner: Owner(2), + bytes: 4096 + }, + "R6.15 the new owner, the same download" + ); + assert_eq!( + effects, + vec![Effect::Render], + "re-render only: nothing cancelled, nothing restarted" + ); +} + +#[test] +fn download_owner_adopt_download_outside_downloading_changes_nothing() { + let now = Instant::now(); + let idle = Machine::default(); + let mut consent = Machine::default(); + consent.on(Input::Click { owner: Owner(1) }, now, &CTX); + let remote = Ctx { + remote: true, + ..CTX + }; + let mut starting = Machine::default(); + starting.on(Input::Click { owner: Owner(1) }, now, &remote); + let mut recording = Machine::default(); + recording.on(Input::Click { owner: Owner(1) }, now, &remote); + recording.on(Input::CaptureStarted, now, &remote); + let mut transcribing = Machine::default(); + transcribing.on(Input::Click { owner: Owner(1) }, now, &remote); + transcribing.on(Input::CaptureStarted, now, &remote); + transcribing.on(Input::Click { owner: Owner(1) }, now, &remote); + + for (name, mut machine) in [ + ("idle", idle), + ("consent", consent), + ("starting", starting), + ("recording", recording), + ("transcribing", transcribing), + ] { + let before = machine.phase().clone(); + let effects = machine.on(Input::AdoptDownload { owner: Owner(2) }, now, &CTX); + assert_eq!(machine.phase(), &before, "{name}: the phase is unchanged"); + assert!(effects.is_empty(), "{name}: no effect: {effects:?}"); + } +} diff --git a/crates/voice_input/src/entity_hardening_tests.rs b/crates/voice_input/src/entity_hardening_tests.rs new file mode 100644 index 0000000..67e1576 --- /dev/null +++ b/crates/voice_input/src/entity_hardening_tests.rs @@ -0,0 +1,492 @@ +// Story 044, task 4.1 fix round 1: the `VoiceInput` entity, hardened after the tech review +// (.draft/reports/4.1-tech-review-rust-gpui.md, findings 1-5). Mounted in the crate root under +// `cfg(test)`. Validation filter: `entity` (every test path is `entity_hardening_tests::…`). +// +// The fakes repeat entity_tests.rs's: that file is frozen, and its fakes are private to it. +// +// Hostile halves: +// - a composer closed while it owns the machine (consent strip or a toggle recording) neither +// blocks the other composers nor keeps the microphone open, and inserts nothing; +// - turning the feature off mid-recording closes the microphone and inserts nothing; +// - a packaged model whose verdict is not cached is never hashed inside `handle` (the UI thread), +// yet the machine still gets its answer once the background hash returns; +// - a plain-http URL is refused whenever a proxy is configured (client or environment), and +// allowed when none is or over https; +// - a press held behind a hash keeps the time it arrived, so a hold stays a hold. + +use std::{ + path::{Path, PathBuf}, + sync::{ + Arc, Mutex, + atomic::{AtomicUsize, Ordering}, + }, + time::{Duration, Instant}, +}; + +use gpui::{AppContext as _, Entity, EntityId, TestAppContext, UpdateGlobal as _}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use settings::SettingsStore; +use url::Url; + +use crate::capture::{ActiveCapture, CaptureBackend, CaptureEvent, Ended, Recording}; +use crate::models::{VoiceModel, spec}; +use crate::remote::proxy_refusal; +use crate::state::{Input, Owner}; +use crate::{ViewState, VoiceEvent, VoiceInput, init_in}; + +#[derive(Default)] +struct Mic { + starts: AtomicUsize, + stops: AtomicUsize, + cancels: AtomicUsize, +} + +struct Backend(Arc); +struct Active(Arc); + +impl CaptureBackend for Backend { + fn start( + &self, + _device: Option, + _events: smol::channel::Sender, + ) -> anyhow::Result> { + self.0.starts.fetch_add(1, Ordering::SeqCst); + Ok(Box::new(Active(self.0.clone()))) + } +} + +impl ActiveCapture for Active { + fn stop(self: Box) -> Recording { + self.0.stops.fetch_add(1, Ordering::SeqCst); + let samples = (0..48_000).map(|i| 0.2 * (i as f32 * 0.05).sin()).collect(); + Recording { + samples, + sample_rate: 48_000, + ended: Ended::User, + } + } + fn cancel(self: Box) { + self.0.cancels.fetch_add(1, Ordering::SeqCst); + } +} + +struct Harness { + vi: Entity, + mic: Arc, + requests: Arc, + events: Arc>>, + models: PathBuf, + _tmp: tempfile::TempDir, +} + +impl Harness { + fn send(&self, input: Input, cx: &mut TestAppContext) { + self.vi.update(cx, |vi, cx| vi.handle(input, cx)); + } + fn view(&self, owner: EntityId, cx: &mut TestAppContext) -> ViewState { + self.vi.read_with(cx, |vi, _| vi.view_state(owner)) + } + fn notices(&self, owner: EntityId) -> Vec { + self.events + .lock() + .expect("lock") + .iter() + .filter_map(|event| match event { + VoiceEvent::Notify { owner: o, message } if *o == owner => Some(message.clone()), + _ => None, + }) + .collect() + } + fn inserts(&self) -> usize { + self.events + .lock() + .expect("lock") + .iter() + .filter(|event| matches!(event, VoiceEvent::Insert { .. })) + .count() + } + fn register(&self, owner: &Entity<()>, cx: &mut TestAppContext) { + let id = owner.entity_id(); + let weak = owner.downgrade().into(); + self.vi + .update(cx, |vi, _| vi.register_owner(Owner(id.as_u64()), weak)); + } +} + +fn set_settings(json: &str, cx: &mut TestAppContext) { + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(json, cx) + .result() + .expect("settings parse"); + }) + }); +} + +fn setup(settings_json: &str, cx: &mut TestAppContext) -> Harness { + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); + set_settings(settings_json, cx); + let tmp = tempfile::tempdir().expect("tempdir"); + let models = tmp.path().join("voice").join("models"); + let mic = Arc::new(Mic::default()); + let requests = Arc::new(AtomicUsize::new(0)); + let counter = requests.clone(); + let http: Arc = FakeHttpClient::create(move |_| { + counter.fetch_add(1, Ordering::SeqCst); + async move { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from(r#"{"text": "words"}"#))?) + } + }); + let backend: Arc = Arc::new(Backend(mic.clone())); + cx.update(|cx| init_in(http, models.clone(), backend, cx)); + let vi = cx.update(|cx| VoiceInput::global(cx)); + let events: Arc>> = Arc::default(); + let sink = events.clone(); + cx.update(|cx| { + cx.subscribe(&vi, move |_, event: &VoiceEvent, _| { + sink.lock().expect("lock").push(event.clone()) + }) + .detach() + }); + cx.run_until_parked(); + Harness { + vi, + mic, + requests, + events, + models, + _tmp: tmp, + } +} + +fn o(id: EntityId) -> Owner { + Owner(id.as_u64()) +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +fn place_sparse_model(dir: &Path, model: VoiceModel) -> PathBuf { + std::fs::create_dir_all(dir).expect("dir"); + let s = spec(model); + let path = dir.join(s.file); + std::fs::File::create(&path) + .and_then(|f| f.set_len(s.size)) + .expect("sparse model"); + path +} + +const REMOTE: &str = r#"{ "voice_input": { "remote": { "enabled": true, + "url": "http://127.0.0.1:9/v1/audio/transcriptions", "api_key_env": "ZEO_VOICE_TEST_UNSET_044_9F3C" } } }"#; + +// ── finding 1: a closed owner does not wedge the machine ─────────────────── + +#[gpui::test] +async fn entity_hardening_closed_owner_in_consent_does_not_block_others(cx: &mut TestAppContext) { + let h = setup("{}", cx); + let closed = cx.new(|_| ()); + let other = cx.new(|_| ()); + let (a, b) = (closed.entity_id(), other.entity_id()); + h.register(&closed, cx); + h.send(Input::Click { owner: o(a) }, cx); + cx.run_until_parked(); + assert!( + matches!(h.view(b, cx), ViewState::Busy), + "precondition: A owns the consent strip" + ); + + drop(closed); + cx.run_until_parked(); + h.send(Input::Click { owner: o(b) }, cx); + cx.run_until_parked(); + assert!( + matches!(h.view(b, cx), ViewState::AwaitingConsent { .. }), + "B gets its own consent strip: {:?}", + h.view(b, cx) + ); + assert!( + h.notices(b).is_empty(), + "B is not told busy: {:?}", + h.notices(b) + ); + assert_eq!( + h.requests.load(Ordering::SeqCst), + 0, + "R6.9 nothing downloaded" + ); +} + +#[gpui::test] +async fn entity_hardening_closed_owner_while_recording_closes_the_mic_within_a_second( + cx: &mut TestAppContext, +) { + let h = setup(REMOTE, cx); + let closed = cx.new(|_| ()); + let a = closed.entity_id(); + h.register(&closed, cx); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "recording", |cx| { + matches!(h.view(a, cx), ViewState::Recording { .. }) + }); + + drop(closed); + cx.run_until_parked(); + cx.executor().advance_clock(Duration::from_millis(1_100)); + cx.run_until_parked(); + assert_eq!( + h.mic.cancels.load(Ordering::SeqCst), + 1, + "the microphone is closed" + ); + assert_eq!( + h.mic.stops.load(Ordering::SeqCst), + 0, + "nothing is transcribed" + ); + assert_eq!(h.inserts(), 0); + assert_eq!( + h.requests.load(Ordering::SeqCst), + 0, + "no audio leaves the machine" + ); + let other = cx.new(|_| ()).entity_id(); + assert!( + matches!(h.view(other, cx), ViewState::Idle { .. }), + "the others are free again: {:?}", + h.view(other, cx) + ); +} + +#[gpui::test] +async fn entity_hardening_closed_owner_while_recording_is_cancelled_by_another_click( + cx: &mut TestAppContext, +) { + let h = setup(REMOTE, cx); + let closed = cx.new(|_| ()); + let other = cx.new(|_| ()); + let (a, b) = (closed.entity_id(), other.entity_id()); + h.register(&closed, cx); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "recording", |cx| { + matches!(h.view(a, cx), ViewState::Recording { .. }) + }); + + drop(closed); + cx.run_until_parked(); + h.send(Input::Click { owner: o(b) }, cx); + wait_until(cx, "B recording", |cx| { + matches!(h.view(b, cx), ViewState::Recording { .. }) + }); + assert_eq!( + h.mic.cancels.load(Ordering::SeqCst), + 1, + "A's recording is dropped" + ); + assert!( + h.notices(b).is_empty(), + "B is not told busy: {:?}", + h.notices(b) + ); +} + +// ── finding 2: disabling mid-recording; the backend is the one at start ──── + +#[gpui::test] +async fn entity_hardening_disabling_mid_recording_closes_the_mic(cx: &mut TestAppContext) { + let h = setup(REMOTE, cx); + let a = cx.new(|_| ()).entity_id(); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "recording", |cx| { + matches!(h.view(a, cx), ViewState::Recording { .. }) + }); + + set_settings(r#"{ "voice_input": { "enabled": false } }"#, cx); + cx.run_until_parked(); + assert_eq!( + h.mic.cancels.load(Ordering::SeqCst), + 1, + "the microphone is closed" + ); + assert_eq!(h.mic.stops.load(Ordering::SeqCst), 0); + assert_eq!( + h.inserts(), + 0, + "nothing reaches a composer whose feature is off" + ); + assert_eq!(h.requests.load(Ordering::SeqCst), 0); + assert!(matches!(h.view(a, cx), ViewState::Hidden)); +} + +#[gpui::test] +async fn entity_hardening_backend_is_the_one_chosen_at_recording_start(cx: &mut TestAppContext) { + let h = setup(REMOTE, cx); + let a = cx.new(|_| ()).entity_id(); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "recording", |cx| { + matches!(h.view(a, cx), ViewState::Recording { .. }) + }); + + // Switched to local (with no model) while recording: the audio recorded under the remote + // setting still goes to the remote service it was recorded for. + set_settings("{}", cx); + cx.run_until_parked(); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "the insert", |_| h.inserts() == 1); + assert_eq!(h.requests.load(Ordering::SeqCst), 1); +} + +// ── findings 3 and 4: no hash on the UI path; an isolated system dir ─────── + +#[gpui::test] +async fn entity_hardening_cold_packaged_model_is_hashed_off_the_ui_path(cx: &mut TestAppContext) { + let h = setup(r#"{ "voice_input": { "model": "tiny" } }"#, cx); + // `init_in`'s packaged-model directory is `/system`. The model is placed after + // init, so the init warm-up found nothing and the verdict cache is cold. + place_sparse_model(&h.models.join("system"), VoiceModel::Tiny); + let a = cx.new(|_| ()).entity_id(); + + // `handle` runs on the UI thread. Counted inside the same update, before the executor runs + // anything else: a hash computed here would be a hash computed on the UI thread. + let hashed_in_handle = h.vi.update(cx, |vi, cx| { + vi.handle(Input::Click { owner: o(a) }, cx); + vi.hash_cache().hashes_computed() + }); + assert_eq!(hashed_in_handle, 0, "no SHA-256 inside handle"); + + // The background hash then answers: zeros are not the pinned file, no downloaded copy + // exists, so the machine asks for consent. + wait_until(cx, "the consent strip", |cx| { + matches!(h.view(a, cx), ViewState::AwaitingConsent { .. }) + }); + let hashed = + h.vi.read_with(cx, |vi, _| vi.hash_cache().hashes_computed()); + assert_eq!( + hashed, 1, + "the packaged file was hashed once, in the background" + ); + assert_eq!(h.mic.starts.load(Ordering::SeqCst), 0); +} + +#[gpui::test] +async fn entity_hardening_init_in_reads_its_own_system_dir(cx: &mut TestAppContext) { + // A valid packaged model cannot be faked (it must hash to the pinned value), so this proves + // the seam the other way: a size-correct file in `/system` is looked at (and + // hashed), which it never is when the entity reads `/usr/share/whisper-models`. + let h = setup(r#"{ "voice_input": { "model": "tiny" } }"#, cx); + place_sparse_model(&h.models.join("system"), VoiceModel::Tiny); + let a = cx.new(|_| ()).entity_id(); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "the consent strip", |cx| { + matches!(h.view(a, cx), ViewState::AwaitingConsent { .. }) + }); + let hashed = + h.vi.read_with(cx, |vi, _| vi.hash_cache().hashes_computed()); + assert_eq!(hashed, 1, "the isolated system dir was consulted"); +} + +// ── finding 5: a plain-http URL and a proxy ───────────────────────────────── + +fn url(raw: &str) -> Url { + Url::parse(raw).expect("url") +} + +const NO_PROXY_ENV: [Option; 6] = [None, None, None, None, None, None]; + +fn env_with(name: &str, value: &str) -> [Option; 6] { + crate::remote::PROXY_ENV_VARS.map(|var| (var == name).then(|| value.to_string())) +} + +#[test] +fn entity_hardening_plain_http_through_a_proxy_is_refused() { + let proxy = url("http://proxy.corp:3128"); + let loopback = url("http://127.0.0.1:9/v1/t"); + let refusal = proxy_refusal(&loopback, Some(&proxy), &NO_PROXY_ENV) + .expect("refused when the client has a proxy"); + assert_eq!( + refusal, + "The remote voice URL uses plain http while a proxy is configured; use https" + ); + // reqwest reads these itself, even when Zed could not parse them into `proxy()`: a + // scheme-less `10.0.0.1:3128` is one such value. + for name in crate::remote::PROXY_ENV_VARS { + assert!( + proxy_refusal(&loopback, None, &env_with(name, "10.0.0.1:3128")).is_some(), + "{name} set" + ); + } + assert!( + proxy_refusal(&url("http://localhost:8080/t"), Some(&proxy), &NO_PROXY_ENV).is_some(), + "NO_PROXY is not consulted: reqwest's matching differs from any copy of it" + ); +} + +#[test] +fn entity_hardening_plain_http_without_a_proxy_or_over_https_is_allowed() { + let proxy = url("http://proxy.corp:3128"); + let loopback = url("http://127.0.0.1:9/v1/t"); + assert_eq!( + proxy_refusal(&loopback, None, &NO_PROXY_ENV), + None, + "no proxy" + ); + assert_eq!( + proxy_refusal(&loopback, None, &env_with("HTTP_PROXY", " ")), + None, + "a blank variable is no proxy" + ); + assert_eq!( + proxy_refusal( + &url("https://stt.example/t"), + Some(&proxy), + &env_with("ALL_PROXY", "p:1") + ), + None, + "https is tunnelled, never readable by the proxy" + ); +} + +// ── re-review R2: a held press keeps its arrival time ────────────────────── + +#[gpui::test] +async fn entity_hardening_press_held_behind_a_hash_keeps_its_press_time(cx: &mut TestAppContext) { + let h = setup(r#"{ "voice_input": { "model": "tiny" } }"#, cx); + // A packaged file that will fail its hash (cold cache, so the press is held) and a + // downloaded copy, so the replayed press finds a model and records. + place_sparse_model(&h.models.join("system"), VoiceModel::Tiny); + place_sparse_model(&h.models, VoiceModel::Tiny); + let a = cx.new(|_| ()).entity_id(); + + h.send(Input::ShortcutDown { owner: o(a) }, cx); + // Held for longer than the tap window before the executor even runs the hash. + std::thread::sleep(Duration::from_millis(450)); + wait_until(cx, "recording", |cx| { + matches!(h.view(a, cx), ViewState::Recording { .. }) + }); + h.send(Input::ShortcutUp { owner: o(a) }, cx); + cx.run_until_parked(); + assert_eq!( + h.mic.stops.load(Ordering::SeqCst), + 1, + "a 450 ms press is a hold (R3.1): its release stops the recording" + ); +} diff --git a/crates/voice_input/src/entity_tests.rs b/crates/voice_input/src/entity_tests.rs new file mode 100644 index 0000000..03dc65b --- /dev/null +++ b/crates/voice_input/src/entity_tests.rs @@ -0,0 +1,753 @@ +// Story 044, task 4.1 — the `VoiceInput` entity runs the machine's effects (R2.1, R2.6, R4.6, R5.1, +// R6.1, R6.5, R6.6, R6.7, R6.9, R7.1, R7.3, R8.2, R8.3, R8.4, R8.5, R8.6, R8.7, R8.8, R9.1, R9.2, +// R10.1, R10.3). Wired in the crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod entity_tests; +// +// Validation filter: `entity` (every test path is `entity_tests::…`). +// +// Contract: design.md C1.10 (`handle`, `view_state`, `VoiceEvent`), C1.9 (`Input`). +// ASSUMED TEST SEAMS (not in design.md) — a test must never open the operator's microphone nor +// write into the real `paths::data_dir()` (a process-wide OnceLock): +// - `crate::init_in(http: Arc, models_dir: PathBuf, +// capture: Arc, cx: &mut App)`; +// `init(http, cx)` is `init_in(http, models::models_dir(), Arc::new(capture::Cpal), cx)`. +// - `capture::CaptureBackend: Send + Sync` with `fn start(&self, device: Option, +// events: smol::channel::Sender) -> anyhow::Result>`, and +// `capture::ActiveCapture: Send` with `fn stop(self: Box) -> Recording` and +// `fn cancel(self: Box)` (`Recorder` implements it). +// - `VoiceInput::global(cx: &App) -> Entity`. +// - `state::Owner(pub u64)`, built from `EntityId::as_u64()`; `VoiceEvent: Clone + Debug`. +// - `ViewState::AwaitingConsent { model: VoiceModel, size: String-like }`, +// `ViewState::Idle { remote_host: Option }`. +// +// Hostile halves: nothing is downloaded before consent AND exactly one pinned GET after it; remote +// mode never asks for a model AND local mode never opens a connection, even when the model fails to +// load; a key in the named env var is sent as a bearer AND an unset one sends no header; a remote +// failure is shown AND never falls back to local; a recording too short or too quiet sends nothing. + +use std::{ + path::{Path, PathBuf}, + str::FromStr as _, + sync::{ + Arc, Mutex, + atomic::{AtomicUsize, Ordering}, + }, + time::{Duration, Instant}, +}; + +use gpui::{AppContext as _, Entity, EntityId, TestAppContext, UpdateGlobal as _}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Method, Response}; +use settings::SettingsStore; + +use crate::capture::{ActiveCapture, CaptureBackend, CaptureEvent, Ended, Recording}; +use crate::models::{VoiceModel, spec}; +use crate::state::{Input, Owner}; +use crate::{ViewState, VoiceEvent, VoiceInput, init_in}; + +// ── fakes ─────────────────────────────────────────────────────────────────── + +#[derive(Default)] +struct CaptureState { + devices: Mutex>>, + stops: AtomicUsize, + cancels: AtomicUsize, + fail_with: Mutex>, + samples: Mutex<(Vec, u32)>, + events: Mutex>>, +} + +struct FakeBackend(Arc); +struct FakeActive(Arc); + +impl CaptureBackend for FakeBackend { + fn start( + &self, + device: Option, + events: smol::channel::Sender, + ) -> anyhow::Result> { + self.0 + .devices + .lock() + .expect("lock") + .push(device.map(|d| format!("{d:?}"))); + if let Some(reason) = self.0.fail_with.lock().expect("lock").clone() { + anyhow::bail!(reason); + } + *self.0.events.lock().expect("lock") = Some(events); + Ok(Box::new(FakeActive(self.0.clone()))) + } +} + +impl ActiveCapture for FakeActive { + fn stop(self: Box) -> Recording { + self.0.stops.fetch_add(1, Ordering::SeqCst); + let (samples, sample_rate) = self.0.samples.lock().expect("lock").clone(); + Recording { + samples, + sample_rate, + ended: Ended::User, + } + } + fn cancel(self: Box) { + self.0.cancels.fetch_add(1, Ordering::SeqCst); + } +} + +fn speech(seconds: f32, rate: u32) -> (Vec, u32) { + let n = (seconds * rate as f32) as usize; + ( + (0..n).map(|i| 0.2 * (i as f32 * 0.05).sin()).collect(), + rate, + ) +} + +#[derive(Clone, Debug)] +struct Seen { + method: Method, + uri: String, + authorization: Option, +} + +type Requests = Arc>>; + +fn http(status: u16, answer: &'static str) -> (Arc, Requests) { + let requests: Requests = Arc::default(); + let log = requests.clone(); + let client = FakeHttpClient::create(move |req| { + log.lock().expect("lock").push(Seen { + method: req.method().clone(), + uri: req.uri().to_string(), + authorization: req + .headers() + .get("authorization") + .and_then(|v| v.to_str().ok()) + .map(str::to_string), + }); + async move { + Ok(Response::builder() + .status(status) + .body(AsyncBody::from(answer))?) + } + }); + (client, requests) +} + +// ── harness ───────────────────────────────────────────────────────────────── + +struct Harness { + vi: Entity, + capture: Arc, + requests: Requests, + events: Arc>>, + models: PathBuf, + _tmp: tempfile::TempDir, + owners: Vec>, +} + +impl Harness { + fn owner(&self, n: usize) -> EntityId { + self.owners[n].entity_id() + } + fn send(&self, input: Input, cx: &mut TestAppContext) { + self.vi.update(cx, |vi, cx| vi.handle(input, cx)); + } + fn view(&self, owner: EntityId, cx: &mut TestAppContext) -> ViewState { + self.vi.read_with(cx, |vi, _| vi.view_state(owner)) + } + fn notices(&self, owner: EntityId) -> Vec { + self.events + .lock() + .expect("lock") + .iter() + .filter_map(|e| match e { + VoiceEvent::Notify { owner: o, message } if *o == owner => { + Some(message.to_string()) + } + _ => None, + }) + .collect() + } + fn inserts(&self) -> Vec<(EntityId, String)> { + self.events + .lock() + .expect("lock") + .iter() + .filter_map(|e| match e { + VoiceEvent::Insert { owner, text } => Some((*owner, text.to_string())), + _ => None, + }) + .collect() + } + fn requests(&self) -> Vec { + self.requests.lock().expect("lock").clone() + } +} + +fn setup( + settings_json: &str, + http_client: (Arc, Requests), + cx: &mut TestAppContext, +) -> Harness { + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(settings_json, cx) + .result() + .expect("settings parse"); + }) + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let models = tmp.path().join("voice").join("models"); + let capture = Arc::new(CaptureState::default()); + *capture.samples.lock().expect("lock") = speech(1.0, 48_000); + let (client, requests) = http_client; + let backend: Arc = Arc::new(FakeBackend(capture.clone())); + cx.update(|cx| init_in(client, models.clone(), backend, cx)); + let vi = cx.update(|cx| VoiceInput::global(cx)); + let events: Arc>> = Arc::default(); + let sink = events.clone(); + cx.update(|cx| { + cx.subscribe(&vi, move |_, event: &VoiceEvent, _| { + sink.lock().expect("lock").push(event.clone()) + }) + .detach() + }); + let owners = vec![cx.new(|_| ()), cx.new(|_| ())]; + Harness { + vi, + capture, + requests, + events, + models, + _tmp: tmp, + owners, + } +} + +fn o(id: EntityId) -> Owner { + Owner(id.as_u64()) +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +fn record_and_stop(h: &Harness, owner: EntityId, cx: &mut TestAppContext) { + h.send(Input::Click { owner: o(owner) }, cx); + wait_until(cx, "the capture to start", |_| { + h.capture.devices.lock().expect("lock").len() == 1 + }); + wait_until(cx, "recording", |cx| { + matches!(h.view(owner, cx), ViewState::Recording { .. }) + }); + h.send(Input::Click { owner: o(owner) }, cx); +} + +fn place_sparse_model(dir: &Path, model: VoiceModel) -> PathBuf { + std::fs::create_dir_all(dir).expect("dir"); + let s = spec(model); + let path = dir.join(s.file); + std::fs::File::create(&path) + .and_then(|f| f.set_len(s.size)) + .expect("sparse model"); + path +} + +const LOCAL_TINY: &str = r#"{ "voice_input": { "model": "tiny" } }"#; +const REMOTE: &str = r#"{ "voice_input": { "remote": { "enabled": true, + "url": "http://127.0.0.1:9/v1/audio/transcriptions", "api_key_env": "ZEO_VOICE_TEST_UNSET_044_9F3C" } } }"#; + +// ── visibility and tooltip data ───────────────────────────────────────────── + +#[gpui::test] +async fn entity_hidden_when_disabled(cx: &mut TestAppContext) { + let h = setup( + r#"{ "voice_input": { "enabled": false } }"#, + http(200, "{}"), + cx, + ); + assert!(matches!(h.view(h.owner(0), cx), ViewState::Hidden), "R1.2"); + h.send( + Input::Click { + owner: o(h.owner(0)), + }, + cx, + ); + cx.run_until_parked(); + assert!( + h.capture.devices.lock().expect("lock").is_empty(), + "nothing starts when disabled" + ); +} + +#[gpui::test] +async fn entity_idle_names_the_remote_host_only_in_remote_mode(cx: &mut TestAppContext) { + let h = setup( + r#"{ "voice_input": { "remote": { "enabled": true, "url": "https://stt.example/v1/t" } } }"#, + http(200, "{}"), + cx, + ); + match h.view(h.owner(0), cx) { + ViewState::Idle { remote_host } => { + assert_eq!(remote_host.as_deref(), Some("stt.example"), "R8.3") + } + other => panic!("{other:?}"), + } + let h = setup("{}", http(200, "{}"), cx); + assert!(matches!( + h.view(h.owner(0), cx), + ViewState::Idle { remote_host: None } + )); +} + +// ── consent and download ──────────────────────────────────────────────────── + +#[gpui::test] +async fn entity_missing_model_asks_consent_and_fetches_nothing(cx: &mut TestAppContext) { + let h = setup("{}", http(404, ""), cx); + let (a, b) = (h.owner(0), h.owner(1)); + h.send(Input::Click { owner: o(a) }, cx); + cx.run_until_parked(); + match h.view(a, cx) { + ViewState::AwaitingConsent { model, size } => { + assert_eq!(model, VoiceModel::Base); + assert_eq!(size.to_string(), "148 MB", "R6.1 the download size"); + } + other => panic!("R6.1 consent first: {other:?}"), + } + assert!( + h.requests().is_empty(), + "R6.9 nothing downloaded before consent" + ); + assert!( + h.capture.devices.lock().expect("lock").is_empty(), + "R6.1 no recording without a model" + ); + + assert!( + matches!(h.view(b, cx), ViewState::Busy), + "R5.1 another composer sees busy" + ); + h.send(Input::Click { owner: o(b) }, cx); + cx.run_until_parked(); + assert!( + h.notices(b) + .iter() + .any(|m| m == "Voice input is busy in another thread"), + "{:?}", + h.notices(b) + ); +} + +#[gpui::test] +async fn entity_declined_consent_downloads_nothing(cx: &mut TestAppContext) { + let h = setup("{}", http(404, ""), cx); + h.send( + Input::Click { + owner: o(h.owner(0)), + }, + cx, + ); + h.send(Input::ConsentDeclined, cx); + cx.run_until_parked(); + assert!(h.requests().is_empty(), "R6.7"); + assert!(matches!(h.view(h.owner(0), cx), ViewState::Idle { .. })); +} + +#[gpui::test] +async fn entity_consent_fetches_the_pinned_url_once_and_reports_http_failure( + cx: &mut TestAppContext, +) { + let h = setup("{}", http(404, "Not Found"), cx); + let a = h.owner(0); + h.send(Input::Click { owner: o(a) }, cx); + h.send(Input::ConsentGiven, cx); + wait_until(cx, "the failure notice", |_| !h.notices(a).is_empty()); + let requests = h.requests(); + assert_eq!( + requests.len(), + 1, + "exactly one request after consent: {requests:?}" + ); + assert_eq!(requests[0].method, Method::GET); + assert_eq!(requests[0].uri, crate::models::url(spec(VoiceModel::Base))); + assert!( + h.notices(a)[0].contains("base"), + "names the model: {:?}", + h.notices(a) + ); + wait_until(cx, "idle", |cx| { + matches!(h.view(a, cx), ViewState::Idle { .. }) + }); + assert!(!h.models.join("ggml-base.bin").exists(), "R6.6"); +} + +#[gpui::test] +async fn entity_wrong_bytes_are_deleted_and_reported_as_a_checksum_mismatch( + cx: &mut TestAppContext, +) { + let h = setup("{}", http(200, "ten bytes!"), cx); + let a = h.owner(0); + h.send(Input::Click { owner: o(a) }, cx); + h.send(Input::ConsentGiven, cx); + wait_until(cx, "the mismatch notice", |_| !h.notices(a).is_empty()); + let msg = &h.notices(a)[0]; + assert!( + msg.contains("base") && msg.contains("checksum"), + "R6.5: {msg}" + ); + let left: Vec<_> = std::fs::read_dir(&h.models) + .map(|d| d.filter_map(|e| e.ok()).collect()) + .unwrap_or_default(); + assert!(left.is_empty(), "R6.5 deleted: {left:?}"); +} + +// ── local transcription: no network, unreadable model ─────────────────────── + +#[gpui::test] +async fn entity_local_mode_opens_no_connection_and_deletes_an_unreadable_model( + cx: &mut TestAppContext, +) { + let h = setup( + LOCAL_TINY, + http(200, r#"{"text": "must not be asked"}"#), + cx, + ); + let a = h.owner(0); + let model = place_sparse_model(&h.models, VoiceModel::Tiny); + record_and_stop(&h, a, cx); + wait_until(cx, "the unreadable notice", |_| !h.notices(a).is_empty()); + let msg = h.notices(a).join(" | "); + assert!( + msg.contains("unreadable") && msg.contains("downloaded again"), + "R7.3: {msg}" + ); + assert!(!model.exists(), "R7.3 the model file is deleted"); + assert!(h.inserts().is_empty(), "R7.3 nothing inserted"); + assert!( + h.requests().is_empty(), + "R7.1 local transcription opens no connection: {:?}", + h.requests() + ); + assert_eq!(h.capture.stops.load(Ordering::SeqCst), 1); +} + +// ── remote transcription ──────────────────────────────────────────────────── + +#[gpui::test] +async fn entity_remote_inserts_the_cleaned_text_and_never_needs_a_model(cx: &mut TestAppContext) { + let h = setup( + REMOTE, + http(200, r#"{"text": " Hello world. [BLANK_AUDIO] "}"#), + cx, + ); + let a = h.owner(0); + record_and_stop(&h, a, cx); + wait_until(cx, "the insert", |_| !h.inserts().is_empty()); + assert_eq!(h.inserts(), vec![(a, "Hello world.".to_string())], "R8.2"); + let requests = h.requests(); + assert_eq!(requests.len(), 1, "{requests:?}"); + assert_eq!(requests[0].method, Method::POST); + assert_eq!( + requests[0].uri, + "http://127.0.0.1:9/v1/audio/transcriptions" + ); + assert_eq!( + requests[0].authorization, None, + "R8.8 unset env var, no header" + ); + assert!( + !requests.iter().any(|r| r.uri.contains("huggingface")), + "R8.7 no model download" + ); + assert!(!h.models.join("ggml-base.bin").exists()); +} + +#[gpui::test] +async fn entity_remote_sends_the_named_env_var_as_bearer(cx: &mut TestAppContext) { + // PATH is always set for cargo test, so no env var has to be written (set_var is unsafe). + let Ok(value) = std::env::var("PATH") else { + return; + }; + if !value.bytes().all(|b| (0x20..0x7f).contains(&b)) { + return; + } + let settings = r#"{ "voice_input": { "remote": { "enabled": true, + "url": "http://127.0.0.1:9/v1/audio/transcriptions", "api_key_env": "PATH" } } }"#; + let h = setup(settings, http(200, r#"{"text": "ok"}"#), cx); + record_and_stop(&h, h.owner(0), cx); + wait_until(cx, "the request", |_| !h.requests().is_empty()); + assert_eq!( + h.requests()[0].authorization, + Some(format!("Bearer {value}")), + "R8.5" + ); +} + +#[gpui::test] +async fn entity_remote_failure_is_shown_without_local_fallback(cx: &mut TestAppContext) { + let h = setup(REMOTE, http(500, "upstream exploded"), cx); + let a = h.owner(0); + record_and_stop(&h, a, cx); + wait_until(cx, "the failure notice", |_| !h.notices(a).is_empty()); + assert!( + h.notices(a).iter().any(|m| m.contains("500")), + "R8.6: {:?}", + h.notices(a) + ); + assert!(h.inserts().is_empty()); + assert_eq!(h.requests().len(), 1, "no retry, no model download"); + wait_until(cx, "idle", |cx| { + matches!(h.view(a, cx), ViewState::Idle { .. }) + }); +} + +#[gpui::test] +async fn entity_remote_plain_http_to_a_remote_host_sends_nothing(cx: &mut TestAppContext) { + let settings = + r#"{ "voice_input": { "remote": { "enabled": true, "url": "http://stt.example/v1/t" } } }"#; + let h = setup(settings, http(200, r#"{"text": "leak"}"#), cx); + let a = h.owner(0); + record_and_stop(&h, a, cx); + wait_until(cx, "the refusal", |_| !h.notices(a).is_empty()); + assert!( + h.notices(a).iter().any(|m| m.contains("https")), + "R8.4: {:?}", + h.notices(a) + ); + assert!(h.requests().is_empty(), "R8.4 nothing sent"); +} + +#[gpui::test] +async fn entity_short_or_silent_recording_sends_nothing(cx: &mut TestAppContext) { + for samples in [speech(0.3, 48_000), (vec![0.0005; 48_000], 48_000)] { + let h = setup(REMOTE, http(200, r#"{"text": "hallucinated"}"#), cx); + *h.capture.samples.lock().expect("lock") = samples; + let a = h.owner(0); + record_and_stop(&h, a, cx); + wait_until(cx, "the notice", |_| !h.notices(a).is_empty()); + assert!( + h.notices(a).iter().any(|m| m == "No speech was recognised"), + "R4.6: {:?}", + h.notices(a) + ); + assert!(h.requests().is_empty() && h.inserts().is_empty()); + } +} + +// ── capture ───────────────────────────────────────────────────────────────── + +#[gpui::test] +async fn entity_uses_the_selected_input_device_or_the_default(cx: &mut TestAppContext) { + let h = setup( + r#"{ "voice_input": { "remote": { "enabled": true, "url": "http://127.0.0.1:9/t" } }, + "audio": { "experimental.input_audio_device": "alsa:proof-input" } }"#, + http(200, r#"{"text":"x"}"#), + cx, + ); + h.send( + Input::Click { + owner: o(h.owner(0)), + }, + cx, + ); + wait_until(cx, "the capture to start", |_| { + !h.capture.devices.lock().expect("lock").is_empty() + }); + let expected = format!( + "{:?}", + cpal::DeviceId::from_str("alsa:proof-input").expect("device id") + ); + assert_eq!( + h.capture.devices.lock().expect("lock")[0], + Some(expected), + "R2.1 the selected device" + ); + + let h = setup(REMOTE, http(200, r#"{"text":"x"}"#), cx); + h.send( + Input::Click { + owner: o(h.owner(0)), + }, + cx, + ); + wait_until(cx, "the capture to start", |_| { + !h.capture.devices.lock().expect("lock").is_empty() + }); + assert_eq!( + h.capture.devices.lock().expect("lock")[0], + None, + "R2.1 the system default" + ); +} + +#[gpui::test] +async fn entity_capture_failure_stays_idle_with_the_reason(cx: &mut TestAppContext) { + let h = setup(REMOTE, http(200, "{}"), cx); + *h.capture.fail_with.lock().expect("lock") = Some("no input device found".into()); + let a = h.owner(0); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "the notice", |_| !h.notices(a).is_empty()); + let msg = &h.notices(a)[0]; + assert!( + msg.starts_with("No microphone available") && msg.contains("no input device found"), + "R9.1: {msg}" + ); + assert!(matches!(h.view(a, cx), ViewState::Idle { .. })); +} + +#[gpui::test] +async fn entity_limit_and_device_error_transcribe_what_was_recorded(cx: &mut TestAppContext) { + for (event, needle) in [ + (CaptureEvent::Limit, "2-minute limit"), + ( + CaptureEvent::DeviceError("stream closed".into()), + "The microphone stopped", + ), + ] { + let h = setup(REMOTE, http(200, r#"{"text": "kept"}"#), cx); + let a = h.owner(0); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "recording", |cx| { + matches!(h.view(a, cx), ViewState::Recording { .. }) + }); + let tx = h + .capture + .events + .lock() + .expect("lock") + .clone() + .expect("events sender"); + tx.try_send(event).expect("send capture event"); + wait_until(cx, "the insert", |_| !h.inserts().is_empty()); + assert!( + h.notices(a).iter().any(|m| m.contains(needle)), + "R2.6/R9.2: {:?}", + h.notices(a) + ); + assert_eq!( + h.inserts(), + vec![(a, "kept".to_string())], + "what was recorded is transcribed" + ); + assert_eq!(h.capture.stops.load(Ordering::SeqCst), 1); + } +} + +#[gpui::test] +async fn entity_cancel_discards_without_any_request(cx: &mut TestAppContext) { + let h = setup(REMOTE, http(200, r#"{"text": "discarded"}"#), cx); + let a = h.owner(0); + h.send(Input::Click { owner: o(a) }, cx); + wait_until(cx, "recording", |cx| { + matches!(h.view(a, cx), ViewState::Recording { .. }) + }); + h.send(Input::Cancel { owner: o(a) }, cx); + cx.run_until_parked(); + assert_eq!(h.capture.cancels.load(Ordering::SeqCst), 1, "R2.5"); + assert_eq!(h.capture.stops.load(Ordering::SeqCst), 0); + assert!(h.requests().is_empty() && h.inserts().is_empty()); +} + +// ── R10.1 / R10.3: one recording line, nothing private in it ──────────────── + +mod log_capture { + use std::sync::{Mutex, OnceLock}; + use std::thread::ThreadId; + + pub struct Capture(pub Mutex>); + impl log::Log for Capture { + fn enabled(&self, _: &log::Metadata) -> bool { + true + } + fn log(&self, r: &log::Record) { + self.0 + .lock() + .expect("lock") + .push((std::thread::current().id(), r.args().to_string())); + } + fn flush(&self) {} + } + pub fn install() -> &'static Capture { + static C: OnceLock<&'static Capture> = OnceLock::new(); + C.get_or_init(|| { + let c: &'static Capture = Box::leak(Box::new(Capture(Mutex::new(Vec::new())))); + log::set_logger(c) + .expect("no other logger is installed in the voice_input test binary"); + log::set_max_level(log::LevelFilter::Trace); + c + }) + } + pub fn mine(c: &Capture) -> Vec { + let me = std::thread::current().id(); + c.0.lock() + .expect("lock") + .iter() + .filter(|(t, _)| *t == me) + .map(|(_, l)| l.clone()) + .collect() + } +} + +#[gpui::test] +async fn entity_logs_one_recording_line_without_private_fields(cx: &mut TestAppContext) { + let capture = log_capture::install(); + let settings = r#"{ "voice_input": { "remote": { "enabled": true, + "url": "http://127.0.0.1:9/v1/t?token=QUERYSECRET", "api_key_env": "ZEO_VOICE_TEST_UNSET_044_9F3C" } } }"#; + let h = setup( + settings, + http(200, r#"{"text": "Transcript words here"}"#), + cx, + ); + record_and_stop(&h, h.owner(0), cx); + wait_until(cx, "the insert", |_| !h.inserts().is_empty()); + let lines: Vec = log_capture::mine(capture) + .into_iter() + .filter(|l| l.starts_with("voice_input ")) + .collect(); + let recording: Vec<&String> = lines + .iter() + .filter(|l| l.contains("mode=") && l.contains("backend=")) + .collect(); + assert_eq!( + recording.len(), + 1, + "R10.1 one line per recording: {lines:?}" + ); + let line = recording[0]; + for key in [ + "mode=toggle", + "backend=remote", + "outcome=", + "audio", + "transcribe", + "model=", + ] { + assert!(line.contains(key), "R10.1 {key}: {line}"); + } + for line in &lines { + assert!(line.starts_with("voice_input event="), "Q14: {line}"); + for leaked in ["Transcript words", "QUERYSECRET", "token="] { + assert!(!line.contains(leaked), "R10.3 {leaked:?} in {line}"); + } + } +} diff --git a/crates/voice_input/src/local.rs b/crates/voice_input/src/local.rs new file mode 100644 index 0000000..d9f1b39 --- /dev/null +++ b/crates/voice_input/src/local.rs @@ -0,0 +1,253 @@ +//! Local transcription with whisper.cpp. Runs on a background thread, never on the UI thread. + +use std::ffi::c_int; +use std::fmt; +use std::path::{Path, PathBuf}; +use std::thread; +use std::time::{Duration, Instant}; + +use anyhow::Context as _; +use whisper_rs::{ + FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters, WhisperState, +}; + +use crate::Language; +use crate::download::model_name; +use crate::models::VoiceModel; + +/// How long a loaded model stays in memory after its last use. +const IDLE_RELEASE: Duration = Duration::from_secs(5 * 60); +const MAX_THREADS: usize = 8; +#[cfg(target_arch = "x86_64")] +const X86_64_V3: &str = "x86-64-v3 (AVX2, FMA, F16C, BMI2)"; + +#[derive(Debug)] +pub enum LocalError { + /// whisper.cpp refused the model file; the caller deletes it so the next use downloads it again. + Unreadable { + model: VoiceModel, + path: PathBuf, + source: anyhow::Error, + }, + Inference(anyhow::Error), +} + +impl fmt::Display for LocalError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + // The model and the path are named by the source's context, so `{:#}` names them once. + match self { + Self::Unreadable { .. } => write!( + f, + "The speech model was unreadable; it will be downloaded again on the next use" + ), + Self::Inference(_) => write!(f, "Local transcription failed"), + } + } +} + +impl std::error::Error for LocalError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Unreadable { source, .. } => Some(source.as_ref()), + Self::Inference(source) => Some(source.as_ref()), + } + } +} + +/// Routes whisper.cpp's and ggml's own output, which they would otherwise write to stderr, +/// through `log`. Idempotent. +pub fn install_logging() { + whisper_rs::install_logging_hooks(); +} + +pub struct LocalTranscriber { + loaded: Option, + last_used: Instant, +} + +struct Loaded { + model: VoiceModel, + context: WhisperContext, + /// Kept between calls: creating a state allocates the KV caches and compute buffers again + /// (and logs them). Dropped after a failed call, so the next one starts from a fresh state. + state: Option, +} + +impl Default for LocalTranscriber { + fn default() -> Self { + Self { + loaded: None, + last_used: Instant::now(), + } + } +} + +impl LocalTranscriber { + /// Transcribes 16 kHz mono f32 samples in [-1, 1] with `model`, loading it from `path` when + /// a different model (or none) is loaded. + pub fn transcribe( + &mut self, + model: VoiceModel, + path: &Path, + language: &Language, + pcm16k: &[f32], + ) -> Result { + self.last_used = Instant::now(); + let name = model_name(model); + let loaded = self.loaded_for(model, path)?; + + let mut state = match loaded.state.take() { + Some(state) => state, + None => loaded + .context + .create_state() + .with_context(|| format!("creating a whisper state for speech model {name}")) + .map_err(LocalError::Inference)?, + }; + + let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 }); + params.set_print_special(false); + params.set_print_progress(false); + params.set_print_realtime(false); + params.set_print_timestamps(false); + params.set_no_context(true); + params.set_suppress_blank(true); + params.set_suppress_nst(true); + params.set_n_threads(thread_count( + thread::available_parallelism().map_or(1, |count| count.get()), + )); + params.set_language(Some(language_param(language))); + + state + .full(params, pcm16k) + .with_context(|| format!("transcribing with speech model {name}")) + .map_err(LocalError::Inference)?; + + // Joined as bytes and decoded once: a multibyte character split across two segments + // would make each half invalid UTF-8 on its own. + let mut bytes = Vec::new(); + for segment in state.as_iter() { + let segment_bytes = segment + .to_bytes() + .with_context(|| { + format!( + "reading segment {} of the transcript from speech model {name}", + segment.segment_index() + ) + }) + .map_err(LocalError::Inference)?; + bytes.extend_from_slice(segment_bytes); + } + let text = String::from_utf8_lossy(&bytes).into_owned(); + + loaded.state = Some(state); + self.last_used = Instant::now(); + Ok(text) + } + + /// Drops the loaded model once it has been unused for five minutes. + pub fn release_if_idle(&mut self, now: Instant) { + if now.saturating_duration_since(self.last_used) < IDLE_RELEASE { + return; + } + if let Some(loaded) = self.loaded.take() { + log::debug!( + "voice_input event=model_released model={}", + model_name(loaded.model) + ); + } + } + + fn loaded_for(&mut self, model: VoiceModel, path: &Path) -> Result<&mut Loaded, LocalError> { + Ok(match self.loaded.take() { + Some(loaded) if loaded.model == model => self.loaded.insert(loaded), + previous => { + // Free the previous model before loading the next, so two never sit in memory at once. + drop(previous); + self.loaded.insert(Loaded { + model, + context: load(model, path)?, + state: None, + }) + } + }) + } +} + +fn load(model: VoiceModel, path: &Path) -> Result { + let started = Instant::now(); + let name = model_name(model); + let context = WhisperContext::new_with_params(path, WhisperContextParameters::default()) + .with_context(|| format!("loading speech model {name} from {}", path.display())) + .map_err(|source| LocalError::Unreadable { + model, + path: path.to_path_buf(), + source, + })?; + log::info!( + "voice_input event=model_loaded model={name} elapsed_ms={}", + started.elapsed().as_millis() + ); + Ok(context) +} + +/// Half the available cores (whisper.cpp gains little from hyper-threads), between 1 and 8. +fn thread_count(available: usize) -> c_int { + let threads = (available / 2).clamp(1, MAX_THREADS); + c_int::try_from(threads).unwrap_or(1) +} + +/// The configured language code when whisper knows it, `None` to let whisper detect it. +/// +/// The settings parser admits any two lowercase letters, and whisper.cpp does not fail on a code +/// it does not know: it decodes with a corrupt prompt. So an unknown code falls back to detection. +pub fn known_language_code(language: &Language) -> Option<&str> { + let Language::Code(code) = language else { + return None; + }; + // Scanning the table rather than calling `whisper_rs::get_lang_id`: whisper.cpp logs an + // unknown code verbatim at error level, and the configured value is never logged. + let known = (0..=whisper_rs::get_lang_max_id()) + .filter_map(whisper_rs::get_lang_str) + .any(|known| known == code); + if known { + Some(code) + } else { + log::warn!("voice_input event=language_unknown"); + None + } +} + +/// What whisper is told to transcribe in: the configured code, or `"auto"` to detect it. +pub fn language_param(language: &Language) -> &str { + known_language_code(language).unwrap_or("auto") +} + +/// Refuses before whisper.cpp runs on a CPU lacking the instruction-set level it was compiled +/// for, which would otherwise die with SIGILL. Only the portable build sets a baseline; a source +/// build is compiled for the user's own `-march`. +pub fn cpu_supported() -> Result<(), &'static str> { + if option_env!("ZEO_VOICE_CPU_BASELINE") == Some("x86-64-v3") { + return x86_64_v3_supported(); + } + Ok(()) +} + +#[cfg(target_arch = "x86_64")] +fn x86_64_v3_supported() -> Result<(), &'static str> { + if std::arch::is_x86_feature_detected!("avx2") + && std::arch::is_x86_feature_detected!("fma") + && std::arch::is_x86_feature_detected!("f16c") + && std::arch::is_x86_feature_detected!("bmi2") + { + Ok(()) + } else { + Err(X86_64_V3) + } +} + +// An x86-64 baseline means nothing to a build for another architecture. +#[cfg(not(target_arch = "x86_64"))] +fn x86_64_v3_supported() -> Result<(), &'static str> { + Ok(()) +} diff --git a/crates/voice_input/src/local_lang_tests.rs b/crates/voice_input/src/local_lang_tests.rs new file mode 100644 index 0000000..a7fe3d5 --- /dev/null +++ b/crates/voice_input/src/local_lang_tests.rs @@ -0,0 +1,24 @@ +// Story 044, task 3.3 fix round 1 (tech review finding 2) — an unknown two-letter code is never +// handed to whisper.cpp, which would decode with a corrupt prompt instead of failing (R7.2, R7.4). + +use crate::Language; +use crate::local::{known_language_code, language_param}; + +#[test] +fn local_lang_unknown_code_falls_back_to_detection() { + assert_eq!(known_language_code(&Language::Code("xx".into())), None); + assert_eq!(language_param(&Language::Code("xx".into())), "auto"); +} + +#[test] +fn local_lang_known_code_is_kept() { + assert_eq!( + known_language_code(&Language::Code("pt".into())), + Some("pt") + ); + assert_eq!( + known_language_code(&Language::Code("en".into())), + Some("en") + ); + assert_eq!(known_language_code(&Language::Auto), None); +} diff --git a/crates/voice_input/src/local_tests.rs b/crates/voice_input/src/local_tests.rs new file mode 100644 index 0000000..cfbf53f --- /dev/null +++ b/crates/voice_input/src/local_tests.rs @@ -0,0 +1,131 @@ +// Story 044, task 3.3 — local transcription with whisper-rs (R7.2, R7.3, R7.4, R11.7). Wired in the +// crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod local_tests; +// +// Validation filter: `local` (every test path is `local_tests::…`). +// +// No real model is used (78 MB download): the readable-model path is proven by the GUI proof (7.3). +// Here whisper.cpp only ever sees files it must refuse. +// +// Contract (design.md C1.7). Assumed — adjust the `use` only: +// - `LocalTranscriber::default()`; `transcribe(&mut self, VoiceModel, &Path, &Language, &[f32]) +// -> anyhow::Result`; a load failure is an error whose chain holds `LocalError::Unreadable` +// (`err.downcast_ref::()`; matched as `Unreadable { .. }`, any variant shape), with context naming the model and the path. +// - `local::cpu_supported() -> Result<(), &'static str>`. +// - `local::language_param(&Language) -> &str` — what is handed to whisper: the code, or "auto". +// +// Hostile halves: a file of the pinned size but not a model (zeros) is Unreadable AND a missing file +// is Unreadable too (both lead to "downloaded again on the next use"); a code is passed as itself AND +// Auto is passed as "auto", never as an empty or English default. + +use std::fs; +use std::time::{Duration, Instant}; + +use crate::Language; +use crate::local::{LocalError, LocalTranscriber, cpu_supported, language_param}; +use crate::models::{VoiceModel, spec}; + +fn speech_like() -> Vec { + (0..16_000).map(|i| 0.1 * (i as f32 * 0.17).sin()).collect() +} + +fn assert_unreadable(err: &anyhow::Error, path: &std::path::Path) { + assert!( + err.chain().any(|e| matches!( + e.downcast_ref::(), + Some(LocalError::Unreadable { .. }) + )), + "a load failure is LocalError::Unreadable: {err:#}" + ); + let shown = format!("{err:#}"); + assert!(shown.contains("tiny"), "names the model: {shown}"); + assert!( + shown.contains(&path.display().to_string()), + "names the path: {shown}" + ); +} + +#[test] +fn local_zero_filled_model_of_the_pinned_size_is_unreadable() { + let tmp = tempfile::tempdir().expect("tempdir"); + let s = spec(VoiceModel::Tiny); + let path = tmp.path().join(s.file); + fs::File::create(&path) + .and_then(|f| f.set_len(s.size)) + .expect("sparse file"); + let mut t = LocalTranscriber::default(); + let err = t + .transcribe(VoiceModel::Tiny, &path, &Language::Auto, &speech_like()) + .map_err(anyhow::Error::from) + .expect_err("zeros are not a ggml model"); + assert_unreadable(&err, &path); +} + +#[test] +fn local_garbage_and_missing_model_are_unreadable() { + let tmp = tempfile::tempdir().expect("tempdir"); + let path = tmp.path().join("ggml-tiny.bin"); + fs::write(&path, b"302 Found").expect("write"); + let mut t = LocalTranscriber::default(); + let err = t + .transcribe( + VoiceModel::Tiny, + &path, + &Language::Code("en".into()), + &speech_like(), + ) + .map_err(anyhow::Error::from) + .expect_err("garbage"); + assert_unreadable(&err, &path); + + let missing = tmp.path().join("absent").join("ggml-tiny.bin"); + let err = t + .transcribe(VoiceModel::Tiny, &missing, &Language::Auto, &speech_like()) + .map_err(anyhow::Error::from) + .expect_err("missing"); + assert_unreadable(&err, &missing); +} + +#[test] +fn local_release_if_idle_with_nothing_loaded_is_harmless() { + let mut t = LocalTranscriber::default(); + t.release_if_idle(Instant::now() + Duration::from_secs(301)); + t.release_if_idle(Instant::now()); +} + +#[test] +fn local_language_is_passed_as_code_or_auto() { + assert_eq!(language_param(&Language::Code("pt".into())), "pt", "R7.2"); + assert_eq!(language_param(&Language::Code("en".into())), "en"); + assert_eq!( + language_param(&Language::Auto), + "auto", + "R7.4 detection, not English" + ); +} + +#[test] +fn local_cpu_check_matches_the_build_baseline() { + match option_env!("ZEO_VOICE_CPU_BASELINE") { + Some("x86-64-v3") => { + #[cfg(target_arch = "x86_64")] + { + let has_v3 = std::arch::is_x86_feature_detected!("avx2") + && std::arch::is_x86_feature_detected!("fma") + && std::arch::is_x86_feature_detected!("f16c") + && std::arch::is_x86_feature_detected!("bmi2"); + match cpu_supported() { + Ok(()) => assert!(has_v3, "accepted a CPU without x86-64-v3"), + Err(level) => { + assert!(!has_v3, "refused a CPU that has x86-64-v3"); + assert_eq!(level, "x86-64-v3 (AVX2, FMA, F16C, BMI2)"); + } + } + } + } + // A source build compiled for the user's own -march: nothing to refuse. + _ => assert_eq!(cpu_supported(), Ok(())), + } +} diff --git a/crates/voice_input/src/models.rs b/crates/voice_input/src/models.rs new file mode 100644 index 0000000..95281a8 --- /dev/null +++ b/crates/voice_input/src/models.rs @@ -0,0 +1,302 @@ +use std::collections::HashMap; +use std::fs::{self, File}; +use std::io::{self, Read as _}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{LazyLock, Mutex}; +use std::time::SystemTime; + +use serde::{Deserialize, Serialize}; +use settings::VoiceModelContent; +use sha2::{Digest as _, Sha256}; + +/// A local Whisper model voice input can download and run. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum VoiceModel { + Tiny, + Base, + Small, +} + +impl From for VoiceModel { + fn from(content: VoiceModelContent) -> Self { + match content { + VoiceModelContent::Tiny => Self::Tiny, + VoiceModelContent::Base => Self::Base, + VoiceModelContent::Small => Self::Small, + } + } +} + +/// The Hugging Face commit every model URL is pinned to, so a file can never change under its hash. +const MODELS_COMMIT: &str = "5359861c739e955e79d9a303bcbc70fb988958b1"; + +/// One downloadable model: where it lives upstream and what it must hash to. +#[derive(Debug, PartialEq, Eq)] +pub struct ModelSpec { + pub model: VoiceModel, + pub file: &'static str, + pub size: u64, + pub sha256: &'static str, +} + +static SPECS: [ModelSpec; 3] = [ + ModelSpec { + model: VoiceModel::Tiny, + file: "ggml-tiny.bin", + size: 77_691_713, + sha256: "be07e048e1e599ad46341c8d2a135645097a538221678b7acdd1b1919c6e1b21", + }, + ModelSpec { + model: VoiceModel::Base, + file: "ggml-base.bin", + size: 147_951_465, + sha256: "60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe", + }, + ModelSpec { + model: VoiceModel::Small, + file: "ggml-small.bin", + size: 487_601_967, + sha256: "1be3a9b2063867b937e64e2ec7483364a79917e157fa98c5d94b5c1fffea987b", + }, +]; + +pub fn spec(model: VoiceModel) -> &'static ModelSpec { + match model { + VoiceModel::Tiny => &SPECS[0], + VoiceModel::Base => &SPECS[1], + VoiceModel::Small => &SPECS[2], + } +} + +pub fn url(spec: &ModelSpec) -> String { + format!( + "https://huggingface.co/ggerganov/whisper.cpp/resolve/{MODELS_COMMIT}/{}", + spec.file + ) +} + +/// Where Zeo downloads models when the package manager did not install them. +pub fn models_dir() -> PathBuf { + paths::data_dir().join("voice").join("models") +} + +/// Where `app-accessibility/whisper-models` installs `ggml-.bin`. +pub fn system_models_dir() -> PathBuf { + PathBuf::from("/usr/share/whisper-models") +} + +#[derive(Debug, PartialEq, Eq)] +pub enum ModelPresence { + Present(PathBuf), + Missing, +} + +/// Size only: the SHA-256 was checked when the file was downloaded (R6.4), and hashing up to +/// 488 MB on every use would cost seconds. +pub fn model_on_disk(dir: &Path, spec: &ModelSpec) -> ModelPresence { + let path = dir.join(spec.file); + match fs::metadata(&path) { + Ok(metadata) if metadata.is_file() && metadata.len() == spec.size => { + ModelPresence::Present(path) + } + Ok(_) => ModelPresence::Missing, + Err(error) if error.kind() == io::ErrorKind::NotFound => ModelPresence::Missing, + Err(error) => { + log::warn!( + "voice_input event=model_stat_failed path={:?} error={:?}", + path.display().to_string(), + error.to_string() + ); + ModelPresence::Missing + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ModelOrigin { + System, + User, +} + +#[derive(Debug, PartialEq, Eq)] +pub struct ResolvedModel { + pub path: PathBuf, + pub origin: ModelOrigin, +} + +/// SHA-256 verdicts for system model files, keyed by path and modification time, so a packaged +/// file is hashed once per change rather than on every recording (R6.11). +#[derive(Default)] +pub struct HashCache { + verdicts: Mutex>, + computed: AtomicUsize, +} + +impl HashCache { + pub fn hashes_computed(&self) -> usize { + self.computed.load(Ordering::Relaxed) + } + + /// The cached verdict for `path` at `modified`, never hashing: `None` when the file has not + /// been hashed since it last changed. + pub fn cached_verdict(&self, path: &Path, modified: SystemTime) -> Option { + let verdicts = self.verdicts.lock().ok()?; + verdicts.get(&(path.to_path_buf(), modified)).copied() + } + + fn digest_matches(&self, path: &Path, modified: SystemTime, expected: &str) -> bool { + let key = (path.to_path_buf(), modified); + if let Ok(verdicts) = self.verdicts.lock() + && let Some(verdict) = verdicts.get(&key) + { + return *verdict; + } + self.computed.fetch_add(1, Ordering::Relaxed); + let verdict = match sha256_hex(path) { + Ok(actual) if actual == expected => true, + Ok(_) => { + log::warn!( + "voice_input event=system_model_mismatch path={:?}", + path.display().to_string() + ); + false + } + Err(error) => { + log::warn!( + "voice_input event=system_model_unreadable path={:?} error={:?}", + path.display().to_string(), + error.to_string() + ); + false + } + }; + match self.verdicts.lock() { + Ok(mut verdicts) => { + verdicts.insert(key, verdict); + } + Err(_) => log::warn!("voice_input event=hash_cache_poisoned"), + } + verdict + } +} + +fn sha256_hex(path: &Path) -> io::Result { + let mut file = File::open(path)?; + let mut hasher = Sha256::new(); + let mut buffer = vec![0u8; 64 * 1024]; + loop { + let read = file.read(&mut buffer)?; + if read == 0 { + break; + } + hasher.update(&buffer[..read]); + } + Ok(hasher + .finalize() + .iter() + .map(|byte| format!("{byte:02x}")) + .collect()) +} + +/// Whether the file at `path` hashes to `spec`'s pinned SHA-256. Reads the whole file, so it runs +/// on a background thread; used to tell a corrupt model from a transient load failure (R7.3). +pub fn matches_pinned_hash(path: &Path, spec: &ModelSpec) -> io::Result { + Ok(sha256_hex(path)? == spec.sha256) +} + +static SYSTEM_HASH_CACHE: LazyLock = LazyLock::new(HashCache::default); + +/// The packaged model first (R6.10), checked by size every time and by SHA-256 once per change +/// (R6.11); then the downloaded one, checked by size (R6.8). +pub fn resolve_model(spec: &ModelSpec) -> Option { + resolve_model_in( + spec, + &system_models_dir(), + &models_dir(), + &SYSTEM_HASH_CACHE, + ) +} + +pub fn resolve_model_in( + spec: &ModelSpec, + system_dir: &Path, + user_dir: &Path, + cache: &HashCache, +) -> Option { + if let ModelPresence::Present(path) = model_on_disk(system_dir, spec) { + match fs::metadata(&path).and_then(|metadata| metadata.modified()) { + Ok(modified) => { + if cache.digest_matches(&path, modified, spec.sha256) { + return Some(ResolvedModel { + path, + origin: ModelOrigin::System, + }); + } + } + Err(error) => log::warn!( + "voice_input event=system_model_unreadable path={:?} error={:?}", + path.display().to_string(), + error.to_string() + ), + } + } + match model_on_disk(user_dir, spec) { + ModelPresence::Present(path) => Some(ResolvedModel { + path, + origin: ModelOrigin::User, + }), + ModelPresence::Missing => None, + } +} + +/// What a lookup that must not hash knows about the configured model. +#[derive(Debug, PartialEq, Eq)] +pub enum CachedLookup { + /// The answer [`resolve_model_in`] would give, from sizes and cached verdicts only. + Resolved(Option), + /// A packaged file of the right size whose SHA-256 has not been checked since it last + /// changed: only [`resolve_model_in`], off the UI thread, can answer. + Unverified, +} + +/// [`resolve_model_in`] without hashing, for the UI thread (R6.11). +pub fn lookup_model_cached( + spec: &ModelSpec, + system_dir: &Path, + user_dir: &Path, + cache: &HashCache, +) -> CachedLookup { + if let ModelPresence::Present(path) = model_on_disk(system_dir, spec) { + match fs::metadata(&path).and_then(|metadata| metadata.modified()) { + Ok(modified) => match cache.cached_verdict(&path, modified) { + Some(true) => { + return CachedLookup::Resolved(Some(ResolvedModel { + path, + origin: ModelOrigin::System, + })); + } + Some(false) => {} + None => return CachedLookup::Unverified, + }, + Err(error) => log::warn!( + "voice_input event=system_model_unreadable path={:?} error={:?}", + path.display().to_string(), + error.to_string() + ), + } + } + CachedLookup::Resolved(match model_on_disk(user_dir, spec) { + ModelPresence::Present(path) => Some(ResolvedModel { + path, + origin: ModelOrigin::User, + }), + ModelPresence::Missing => None, + }) +} + +/// Decimal megabytes, rounded: what the consent prompt shows. +pub fn human_size(bytes: u64) -> String { + format!("{} MB", (bytes + 500_000) / 1_000_000) +} diff --git a/crates/voice_input/src/models_system_tests.rs b/crates/voice_input/src/models_system_tests.rs new file mode 100644 index 0000000..1777c08 --- /dev/null +++ b/crates/voice_input/src/models_system_tests.rs @@ -0,0 +1,204 @@ +// Story 044, task 2.1 (v2) — the system model directory first (R6.10, R6.11). Authored at Run time +// by the orchestrator; wired in the crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod models_system_tests; +// +// Validation filter: `models` (every test path is `models_system_tests::…`). +// +// Contract (design.md C1.2 v2, as built — see the deviation register for 2.1): +// - `resolve_model_in(spec, system_dir, user_dir, cache) -> Option`, the seam behind +// `resolve_model(spec)`, which passes `system_models_dir()`, `models_dir()` and a process-wide cache. +// - `ResolvedModel { path: PathBuf, origin: ModelOrigin }`, `ModelOrigin::{System, User}`. +// - `HashCache::default()` remembers the SHA-256 verdict per `(path, mtime)`; `hashes_computed()` +// counts how many files it actually hashed. +// +// Hostile halves: a system file with the pinned size and SHA-256 is used AND one with the pinned size +// but another SHA-256 is skipped for the user directory; an unchanged file is hashed once AND a file +// rewritten with a new mtime is hashed again; nothing in either directory resolves to None. + +use std::fs; +use std::time::{Duration, SystemTime}; + +use crate::models::{ + HashCache, ModelOrigin, ModelSpec, ResolvedModel, VoiceModel, resolve_model_in, + system_models_dir, +}; + +// Twelve bytes stand in for a model: the rules are size-then-SHA-256, not the model's content. +const CONTENT: &[u8] = b"fake-ggml-01"; +// Any digest but CONTENT's own: the pinned value a tampered or corrupt file would fail. +const WRONG_SHA256: &str = "0000000000000000000000000000000000000000000000000000000000000000"; + +fn fake_spec(sha256: &'static str) -> ModelSpec { + ModelSpec { + model: VoiceModel::Base, + file: "ggml-base.bin", + size: CONTENT.len() as u64, + sha256, + } +} + +fn real_sha256() -> &'static str { + use sha2::{Digest, Sha256}; + let digest = Sha256::digest(CONTENT); + let hex: String = digest.iter().map(|byte| format!("{byte:02x}")).collect(); + Box::leak(hex.into_boxed_str()) +} + +#[test] +fn models_system_dir_is_the_package_location() { + assert_eq!( + system_models_dir(), + std::path::PathBuf::from("/usr/share/whisper-models") + ); +} + +#[test] +fn models_system_file_with_pinned_digest_is_used_first() { + let system = tempfile::tempdir().expect("tempdir"); + let user = tempfile::tempdir().expect("tempdir"); + fs::write(system.path().join("ggml-base.bin"), CONTENT).expect("write system model"); + fs::write(user.path().join("ggml-base.bin"), CONTENT).expect("write user model"); + let spec = fake_spec(real_sha256()); + let cache = HashCache::default(); + + let resolved = resolve_model_in(&spec, system.path(), user.path(), &cache); + + assert_eq!( + resolved, + Some(ResolvedModel { + path: system.path().join("ggml-base.bin"), + origin: ModelOrigin::System, + }), + "R6.10 the packaged model wins over the downloaded one" + ); +} + +#[test] +fn models_system_file_with_wrong_digest_falls_back_to_user_dir() { + let system = tempfile::tempdir().expect("tempdir"); + let user = tempfile::tempdir().expect("tempdir"); + fs::write(system.path().join("ggml-base.bin"), CONTENT).expect("write system model"); + fs::write(user.path().join("ggml-base.bin"), CONTENT).expect("write user model"); + // Same size, another digest: a tampered or corrupt packaged file. + let spec = fake_spec(WRONG_SHA256); + let cache = HashCache::default(); + + let resolved = resolve_model_in(&spec, system.path(), user.path(), &cache); + + assert_eq!( + resolved, + Some(ResolvedModel { + path: user.path().join("ggml-base.bin"), + origin: ModelOrigin::User, + }), + "R6.11 a mismatching system file is skipped" + ); +} + +#[test] +fn models_system_wrong_size_is_skipped_without_hashing() { + let system = tempfile::tempdir().expect("tempdir"); + let user = tempfile::tempdir().expect("tempdir"); + fs::write(system.path().join("ggml-base.bin"), b"short").expect("write system model"); + fs::write(user.path().join("ggml-base.bin"), CONTENT).expect("write user model"); + let spec = fake_spec(real_sha256()); + let cache = HashCache::default(); + + let resolved = resolve_model_in(&spec, system.path(), user.path(), &cache); + + assert_eq!( + resolved.map(|model| model.origin), + Some(ModelOrigin::User), + "a system file of the wrong size is skipped for the user directory" + ); + assert_eq!( + cache.hashes_computed(), + 0, + "the size is checked before any hashing" + ); +} + +#[test] +fn models_system_digest_is_computed_once_per_unchanged_file() { + let system = tempfile::tempdir().expect("tempdir"); + let user = tempfile::tempdir().expect("tempdir"); + let path = system.path().join("ggml-base.bin"); + fs::write(&path, CONTENT).expect("write system model"); + let spec = fake_spec(real_sha256()); + let cache = HashCache::default(); + + for _ in 0..3 { + let resolved = resolve_model_in(&spec, system.path(), user.path(), &cache); + assert_eq!( + resolved.map(|model| model.origin), + Some(ModelOrigin::System) + ); + } + assert_eq!( + cache.hashes_computed(), + 1, + "R6.11 hashed once while unchanged" + ); + + // A rewrite with a new modification time is hashed again. + fs::write(&path, CONTENT).expect("rewrite system model"); + let later = SystemTime::now() + Duration::from_secs(5); + fs::File::options() + .write(true) + .open(&path) + .and_then(|file| file.set_modified(later)) + .expect("bump mtime"); + let resolved = resolve_model_in(&spec, system.path(), user.path(), &cache); + assert_eq!( + resolved.map(|model| model.origin), + Some(ModelOrigin::System) + ); + assert_eq!(cache.hashes_computed(), 2, "a changed file is hashed again"); +} + +#[test] +fn models_wrong_digest_is_hashed_once_too() { + let system = tempfile::tempdir().expect("tempdir"); + let user = tempfile::tempdir().expect("tempdir"); + fs::write(system.path().join("ggml-base.bin"), CONTENT).expect("write system model"); + let spec = fake_spec(WRONG_SHA256); + let cache = HashCache::default(); + + for _ in 0..3 { + assert_eq!( + resolve_model_in(&spec, system.path(), user.path(), &cache), + None + ); + } + assert_eq!( + cache.hashes_computed(), + 1, + "the mismatch is remembered, so its warning is logged once" + ); +} + +#[test] +fn models_user_dir_used_when_system_dir_lacks_the_model() { + let system = tempfile::tempdir().expect("tempdir"); + let user = tempfile::tempdir().expect("tempdir"); + fs::write(user.path().join("ggml-base.bin"), CONTENT).expect("write user model"); + let spec = fake_spec(real_sha256()); + let cache = HashCache::default(); + + let resolved = resolve_model_in(&spec, system.path(), user.path(), &cache); + + assert_eq!( + resolved, + Some(ResolvedModel { + path: user.path().join("ggml-base.bin"), + origin: ModelOrigin::User, + }) + ); + assert_eq!( + cache.hashes_computed(), + 0, + "the user directory is checked by size only (R6.8)" + ); +} diff --git a/crates/voice_input/src/models_tests.rs b/crates/voice_input/src/models_tests.rs new file mode 100644 index 0000000..5942cd3 --- /dev/null +++ b/crates/voice_input/src/models_tests.rs @@ -0,0 +1,150 @@ +// Story 044, task 2.1 — the pinned model table and the on-disk check (R6.1, R6.8). Wired in the +// crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod models_tests; +// +// Validation filter: `models` (every test path is `models_tests::…`). +// +// Contract (design.md C1.2, Data Models). Assumed: `ModelPresence::{Present(PathBuf), Missing}` +// with `PartialEq + Debug`; `spec`, `url`, `models_dir`, `model_on_disk`, `human_size` public in +// `crate::models`. +// +// Hostile halves: a file of exactly the pinned size is Present AND one byte short or one byte long +// is Missing (R6.8); a directory with the model's name is Missing, not Present; the URL carries the +// pinned commit AND never a moving ref (`main`). + +use std::fs; + +use crate::models::{ModelPresence, VoiceModel, human_size, model_on_disk, models_dir, spec, url}; + +const PREFIX: &str = "https://huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/"; + +#[test] +fn models_table_is_pinned_byte_for_byte() { + let rows = [ + ( + VoiceModel::Tiny, + "ggml-tiny.bin", + 77_691_713u64, + "be07e048e1e599ad46341c8d2a135645097a538221678b7acdd1b1919c6e1b21", + ), + ( + VoiceModel::Base, + "ggml-base.bin", + 147_951_465, + "60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe", + ), + ( + VoiceModel::Small, + "ggml-small.bin", + 487_601_967, + "1be3a9b2063867b937e64e2ec7483364a79917e157fa98c5d94b5c1fffea987b", + ), + ]; + for (model, file, size, sha) in rows { + let s = spec(model); + assert_eq!(s.model, model); + assert_eq!(s.file, file); + assert_eq!(s.size, size, "{file}"); + assert_eq!(s.sha256, sha, "{file}"); + assert_eq!(s.sha256.len(), 64); + assert!( + s.sha256 + .chars() + .all(|c| c.is_ascii_hexdigit() && !c.is_ascii_uppercase()) + ); + } +} + +#[test] +fn models_url_pins_the_commit() { + for model in [VoiceModel::Tiny, VoiceModel::Base, VoiceModel::Small] { + let s = spec(model); + let u = url(s); + assert_eq!(u, format!("{PREFIX}{}", s.file)); + assert!( + !u.contains("/main/") && !u.contains("/resolve/main"), + "never a moving ref: {u}" + ); + assert!( + u.starts_with("https://huggingface.co/"), + "R6.1 the named host" + ); + } +} + +#[test] +fn models_dir_is_voice_models_under_the_data_dir() { + let dir = models_dir(); + assert!(dir.ends_with("voice/models"), "{}", dir.display()); + assert!(dir.starts_with(paths::data_dir()), "{}", dir.display()); +} + +#[test] +fn models_exact_size_is_present() { + let tmp = tempfile::tempdir().expect("tempdir"); + let s = spec(VoiceModel::Tiny); + let path = tmp.path().join(s.file); + // A sparse file: the size is what is checked, not the bytes (the hash is checked at download). + fs::File::create(&path) + .and_then(|f| f.set_len(s.size)) + .expect("sparse file"); + assert_eq!(model_on_disk(tmp.path(), s), ModelPresence::Present(path)); +} + +#[test] +fn models_one_byte_short_or_long_is_missing() { + let tmp = tempfile::tempdir().expect("tempdir"); + let s = spec(VoiceModel::Tiny); + let path = tmp.path().join(s.file); + for len in [s.size - 1, s.size + 1, 0] { + fs::File::create(&path) + .and_then(|f| f.set_len(len)) + .expect("sparse file"); + assert_eq!( + model_on_disk(tmp.path(), s), + ModelPresence::Missing, + "R6.8 {len} bytes" + ); + } +} + +#[test] +fn models_absent_file_or_a_directory_is_missing() { + let tmp = tempfile::tempdir().expect("tempdir"); + let s = spec(VoiceModel::Base); + assert_eq!(model_on_disk(tmp.path(), s), ModelPresence::Missing); + fs::create_dir(tmp.path().join(s.file)).expect("dir"); + assert_eq!( + model_on_disk(tmp.path(), s), + ModelPresence::Missing, + "a directory is not a model" + ); + assert_eq!( + model_on_disk(&tmp.path().join("nope"), s), + ModelPresence::Missing + ); +} + +#[test] +fn models_another_models_file_does_not_count() { + let tmp = tempfile::tempdir().expect("tempdir"); + let tiny = spec(VoiceModel::Tiny); + fs::File::create(tmp.path().join(tiny.file)) + .and_then(|f| f.set_len(tiny.size)) + .expect("file"); + assert_eq!( + model_on_disk(tmp.path(), spec(VoiceModel::Base)), + ModelPresence::Missing + ); +} + +#[test] +fn models_human_size_rounds_decimal_megabytes() { + assert_eq!(human_size(77_691_713), "78 MB"); + assert_eq!(human_size(147_951_465), "148 MB"); + assert_eq!(human_size(487_601_967), "488 MB"); + assert_eq!(human_size(1_499_999), "1 MB"); + assert_eq!(human_size(1_500_000), "2 MB"); +} diff --git a/crates/voice_input/src/remote.rs b/crates/voice_input/src/remote.rs new file mode 100644 index 0000000..5927a7e --- /dev/null +++ b/crates/voice_input/src/remote.rs @@ -0,0 +1,435 @@ +//! Remote transcription: an OpenAI-compatible `/audio/transcriptions` endpoint, opt-in. + +use std::cell::Cell; +use std::hash::{BuildHasher as _, Hasher as _, RandomState}; +use std::net::Ipv6Addr; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use anyhow::{Context as _, anyhow, bail}; +use futures::{AsyncReadExt as _, FutureExt as _}; +use http_client::http::HeaderValue; +use http_client::{AsyncBody, HttpClient, HttpRequestExt as _, Method, RedirectPolicy, Request}; +use serde::Deserialize; +use url::{Host, Url}; + +use crate::Language; + +/// How long the service has to answer, body included, before the request fails (R8.6). +pub const REMOTE_TIMEOUT: Duration = Duration::from_secs(60); + +/// At most this much of a non-success answer's body is carried into the error. +const ERROR_BODY_LIMIT: usize = 200; + +/// A success answer larger than this is refused: a two-minute transcript is a few KiB of JSON. +const ANSWER_LIMIT: usize = 1024 * 1024; + +#[derive(Deserialize)] +struct Answer { + text: String, +} + +/// Accepts `https` to any host, and plain `http` only to `localhost`, `127.0.0.0/8` or `::1`. +/// A URL carrying user-info is refused whatever its scheme: the key belongs in the environment. +/// A fragment is refused too: it never reaches the server, but it would reach transport errors. +pub fn validate_url(raw: &str) -> anyhow::Result { + // `raw` is never echoed: it may carry credentials. + let url = Url::parse(raw).context("The remote voice URL is not a valid URL")?; + let scheme = url.scheme(); + let host = url + .host_str() + .filter(|host| !host.is_empty()) + .unwrap_or("no host"); + if !url.username().is_empty() || url.password().is_some() { + bail!( + "The remote voice URL must not carry credentials ({scheme}://…@{host}); \ + the API key is read from the environment variable named by `api_key_env`" + ); + } + if url.fragment().is_some() { + bail!("The remote voice URL must not carry a fragment ({scheme}://{host})"); + } + // `scrub` cuts a query at the first `)`, which the `url` crate leaves unencoded. + if url.query().is_some_and(|query| query.contains(')')) { + bail!("The remote voice URL's query must not contain `)` ({scheme}://{host})"); + } + let loopback = match url.host() { + Some(Host::Domain(domain)) => domain == "localhost", + Some(Host::Ipv4(address)) => address.is_loopback(), + Some(Host::Ipv6(address)) => address == Ipv6Addr::LOCALHOST, + None => false, + }; + match scheme { + "https" => Ok(url), + "http" if loopback => Ok(url), + "http" => bail!( + "The remote voice URL must use https: {scheme}://{host} is not a loopback address" + ), + _ => bail!("The remote voice URL must use https, not {scheme} ({host})"), + } +} + +/// The environment variables reqwest reads a proxy from on its own, whatever Zed's settings say. +pub const PROXY_ENV_VARS: [&str; 6] = [ + "HTTP_PROXY", + "http_proxy", + "HTTPS_PROXY", + "https_proxy", + "ALL_PROXY", + "all_proxy", +]; + +/// The refusal shown when a plain-http `url` would go through a proxy. `validate_url` admits +/// plain http only because loopback traffic never leaves the machine; a proxy breaks that +/// premise, and the audio and the bearer key would reach it unencrypted. +/// +/// `client_proxy` is the HTTP client's own proxy; `proxy_env` holds the values of +/// [`PROXY_ENV_VARS`], which reqwest also reads by itself (even ones Zed could not parse). Any +/// proxy refuses plain http: `NO_PROXY` is deliberately not consulted, because predicting +/// reqwest's exemption rules wrongly would leak, and refusing only costs a switch to https. +pub fn proxy_refusal( + url: &Url, + client_proxy: Option<&Url>, + proxy_env: &[Option], +) -> Option { + if url.scheme() != "http" { + return None; + } + let env_proxy = proxy_env + .iter() + .flatten() + .any(|value| !value.trim().is_empty()); + if client_proxy.is_none() && !env_proxy { + return None; + } + Some("The remote voice URL uses plain http while a proxy is configured; use https".to_string()) +} + +/// The URL as it may be shown or logged: scheme, host, port and path, nothing else (R10.3). +pub fn redacted(url: &Url) -> String { + let host = url.host_str().unwrap_or_default(); + let port = url + .port() + .map(|port| format!(":{port}")) + .unwrap_or_default(); + format!("{}://{host}{port}{}", url.scheme(), url.path()) +} + +/// A random multipart boundary that does not occur anywhere in `wav`. +pub fn new_boundary(wav: &[u8]) -> String { + loop { + // `RandomState` is seeded randomly and every instance gets different keys, which is all a + // boundary needs: it must not collide with the payload, it need not be secret. + let high = RandomState::new().build_hasher().finish(); + let low = RandomState::new().build_hasher().finish(); + let boundary = format!("zeo-voice-{high:016x}{low:016x}"); + if !contains(wav, boundary.as_bytes()) { + return boundary; + } + } +} + +fn contains(haystack: &[u8], needle: &[u8]) -> bool { + haystack + .windows(needle.len()) + .any(|window| window == needle) +} + +/// The `multipart/form-data` body of a transcription request, delimited by `boundary`. +pub fn multipart(boundary: &str, wav: &[u8], model: &str, language: &Language) -> Vec { + let mut body = Vec::with_capacity(wav.len() + 512); + body.extend_from_slice( + format!( + "--{boundary}\r\n\ + Content-Disposition: form-data; name=\"file\"; filename=\"audio.wav\"\r\n\ + Content-Type: audio/wav\r\n\r\n" + ) + .as_bytes(), + ); + body.extend_from_slice(wav); + body.extend_from_slice(b"\r\n"); + + let mut field = |name: &str, value: &str| { + body.extend_from_slice( + format!( + "--{boundary}\r\n\ + Content-Disposition: form-data; name=\"{name}\"\r\n\r\n\ + {value}\r\n" + ) + .as_bytes(), + ); + }; + field("model", model); + field("response_format", "json"); + // Without the field the service detects the language itself (R7.4). + if let Language::Code(code) = language { + field("language", code); + } + + body.extend_from_slice(format!("--{boundary}--\r\n").as_bytes()); + body +} + +/// [`transcribe_within`] with the [`REMOTE_TIMEOUT`] of R8.6. +pub async fn transcribe( + http: Arc, + url: &Url, + key: Option, + body: Vec, + boundary: &str, +) -> anyhow::Result { + transcribe_within(http, url, key, body, boundary, REMOTE_TIMEOUT).await +} + +/// Posts `body` (built by [`multipart`] with `boundary`) to `url` and returns the `text` of the +/// answer. `key`, when present, is sent as a bearer token and nowhere else (R8.5, R8.8, R10.3). +/// +/// The whole exchange, body included, must end within `timeout`. A non-success status is an +/// error carrying the status and at most 200 bytes of the body (R8.6). +pub async fn transcribe_within( + http: Arc, + url: &Url, + key: Option, + body: Vec, + boundary: &str, + timeout: Duration, +) -> anyhow::Result { + let shown = redacted(url); + let log = RemoteLog { + url: shown.clone(), + started: Instant::now(), + status: Cell::new(None), + outcome: Cell::new("cancelled"), + }; + + let request = match build_request(url, key.as_deref(), body, boundary) { + Ok(request) => request, + Err(error) => { + log.outcome.set("request"); + return Err(error.context(format!( + "building the remote transcription request for {shown}" + ))); + } + }; + + // `smol::Timer` because this future runs outside gpui and takes no executor (as in + // `download.rs`); the tests drive it with `futures::executor::block_on`. + #[allow(clippy::disallowed_methods)] + let timer = smol::Timer::after(timeout); + let result = futures::select_biased! { + result = exchange(http.as_ref(), request, &shown, key.as_deref(), &log).fuse() => result, + _ = timer.fuse() => { + log.outcome.set("timeout"); + Err(anyhow!( + "Remote transcription timed out after {} s ({shown})", + timeout.as_secs_f64() + )) + } + }; + if result.is_ok() { + log.outcome.set("ok"); + } + result +} + +fn build_request( + url: &Url, + key: Option<&str>, + body: Vec, + boundary: &str, +) -> anyhow::Result> { + let authorization = match key { + Some(key) => { + // The `http` error does not carry the value, and neither does this one. + let mut value = HeaderValue::from_str(&format!("Bearer {key}")).map_err(|_| { + anyhow!("The remote voice API key is not a valid HTTP header value") + })?; + value.set_sensitive(true); + Some(value) + } + None => None, + }; + Ok(Request::builder() + .method(Method::POST) + .uri(url.as_str()) + // A redirect would carry the bearer token to whatever host it names. + .follow_redirects(RedirectPolicy::NoFollow) + .header( + "Content-Type", + format!("multipart/form-data; boundary={boundary}"), + ) + .when_some(authorization, |builder, value| { + builder.header("Authorization", value) + }) + .body(AsyncBody::from(body))?) +} + +async fn exchange( + http: &dyn HttpClient, + request: Request, + shown: &str, + key: Option<&str>, + log: &RemoteLog, +) -> anyhow::Result { + let mut response = match http.send(request).await { + Ok(response) => response, + Err(error) => { + log.outcome.set("send"); + return Err(anyhow!( + "Remote transcription failed: could not reach {shown}: {}", + scrub(&error, shown) + )); + } + }; + let status = response.status(); + log.status.set(Some(status.as_u16())); + + if !status.is_success() { + log.outcome.set("http"); + // Read past the limit by the key's length, so a key straddling the cut is still whole + // when it is redacted. + let limit = ERROR_BODY_LIMIT + key.map_or(0, str::len); + let mut head = Vec::new(); + let read = response + .body_mut() + .take(limit as u64) + .read_to_end(&mut head) + .await; + // The body only decorates the status error; failing to read it leaves the status alone. + let excerpt = match read { + Ok(_) => body_excerpt(&head, key), + Err(_) => String::new(), + }; + let status = status.as_u16(); + if excerpt.is_empty() { + bail!("Remote transcription failed: HTTP {status} from {shown}"); + } + bail!("Remote transcription failed: HTTP {status} from {shown}: {excerpt}"); + } + + let mut bytes = Vec::new(); + let read = response + .body_mut() + .take(ANSWER_LIMIT as u64 + 1) + .read_to_end(&mut bytes) + .await; + if let Err(error) = read { + log.outcome.set("body"); + return Err(anyhow!( + "Remote transcription failed: reading the answer of {shown}: {}", + scrub(&anyhow::Error::from(error), shown) + )); + } + if bytes.len() > ANSWER_LIMIT { + log.outcome.set("too_large"); + bail!( + "Remote transcription failed: the transcription answer of {shown} is larger than 1 MiB" + ); + } + match serde_json::from_slice::(&bytes) { + Ok(answer) => Ok(answer.text), + Err(error) => { + log.outcome.set("parse"); + Err(anyhow::Error::from(error).context(format!( + "Remote transcription failed: {shown} did not answer with a JSON `text` field" + ))) + } + } +} + +/// At most [`ERROR_BODY_LIMIT`] bytes of `head`, on one line, with no trace of `key` (R10.3): +/// a service may echo the key it refused. +fn body_excerpt(head: &[u8], key: Option<&str>) -> String { + let mut text = String::from_utf8_lossy(head).into_owned(); + let key = key.filter(|key| !key.is_empty()); + if let Some(key) = key { + text = text.replace(key, "[redacted]"); + } + // One error, one log line (Q14): a service must not be able to break or forge a line. + let mut text: String = text + .chars() + .map(|character| { + if character.is_control() { + ' ' + } else { + character + } + }) + .collect(); + let mut end = text.len().min(ERROR_BODY_LIMIT); + while !text.is_char_boundary(end) { + end -= 1; + } + text.truncate(end); + // A key cut by the read limit or by the truncation survives the replace as a prefix of itself + // at the very end; redaction shortens what precedes it, so this can be any echo, not only the + // first. + if let Some(key) = key { + let partial = (1..key.len()) + .rev() + .find(|&length| text.as_bytes().ends_with(&key.as_bytes()[..length])); + if let Some(length) = partial { + let cut = text.len() - length; + if text.is_char_boundary(cut) { + text.truncate(cut); + } + } + } + text.trim().to_owned() +} + +/// The formatted error chain with the query (and any fragment) cut after every occurrence of the +/// redacted URL. Transport errors name the URL they failed on, query included, and reqwest may +/// already have rewritten part of it, so the URL cannot be matched as a whole (R10.3). +fn scrub(error: &anyhow::Error, shown: &str) -> String { + let text = format!("{error:#}"); + let mut scrubbed = String::with_capacity(text.len()); + let mut rest = text.as_str(); + while let Some(at) = rest.find(shown) { + let after = at + shown.len(); + scrubbed.push_str(&rest[..after]); + rest = &rest[after..]; + if rest.starts_with(['?', '#']) { + let end = rest + .find(|character: char| { + character == ')' || character == '"' || character.is_whitespace() + }) + .unwrap_or(rest.len()); + rest = &rest[end..]; + } + } + scrubbed.push_str(rest); + scrubbed +} + +/// Logs exactly one `remote_transcribe` line when the request ends — including when the future is +/// dropped mid-request, reported as `cancelled`. Only the redacted URL appears (R10.3). +struct RemoteLog { + url: String, + started: Instant, + status: Cell>, + outcome: Cell<&'static str>, +} + +impl Drop for RemoteLog { + fn drop(&mut self) { + let outcome = self.outcome.get(); + let level = match outcome { + "ok" | "cancelled" => log::Level::Info, + _ => log::Level::Warn, + }; + let status = self + .status + .get() + .map(|status| status.to_string()) + .unwrap_or_else(|| "none".to_owned()); + log::log!( + level, + "voice_input event=remote_transcribe url={} status={} duration_ms={} outcome={}", + self.url, + status, + self.started.elapsed().as_millis(), + outcome + ); + } +} diff --git a/crates/voice_input/src/remote_host_tests.rs b/crates/voice_input/src/remote_host_tests.rs new file mode 100644 index 0000000..6586f6e --- /dev/null +++ b/crates/voice_input/src/remote_host_tests.rs @@ -0,0 +1,173 @@ +// Story 044, task 5.2 fix round 1 (L2) — the host the button names while audio is in flight +// (R8.3, R8.7). `VoiceInput::remote_host` answers the host of the recording in flight, which is the +// backend it is transcribed by, and the configured host otherwise. +// +// Hostile halves: the host is named while recording AND while transcribing; it stays the +// recording's own after the settings switch to local mid-recording AND it is `None` for a +// recording made in local mode. + +use std::{ + sync::{ + Arc, + atomic::{AtomicUsize, Ordering}, + }, + time::{Duration, Instant}, +}; + +use gpui::{AppContext as _, Entity, EntityId, TestAppContext, UpdateGlobal as _}; +use http_client::{FakeHttpClient, HttpClient}; +use settings::SettingsStore; + +use crate::capture::{ActiveCapture, CaptureBackend, CaptureEvent, Ended, Recording}; +use crate::state::{Input, Owner}; +use crate::{ViewState, VoiceInput, init_in}; + +const REMOTE: &str = r#"{ "voice_input": { "remote": { "enabled": true, + "url": "https://api.groq.com/openai/v1/audio/transcriptions", + "api_key_env": "ZEO_VOICE_TEST_UNSET_044_REMOTE_HOST" } } }"#; + +struct Backend(Arc); +struct Active; + +impl CaptureBackend for Backend { + fn start( + &self, + _device: Option, + _events: smol::channel::Sender, + ) -> anyhow::Result> { + self.0.fetch_add(1, Ordering::SeqCst); + Ok(Box::new(Active)) + } +} + +impl ActiveCapture for Active { + fn stop(self: Box) -> Recording { + let samples = (0..48_000).map(|i| 0.2 * (i as f32 * 0.05).sin()).collect(); + Recording { + samples, + sample_rate: 48_000, + ended: Ended::User, + } + } + fn cancel(self: Box) {} +} + +fn set_settings(json: &str, cx: &mut TestAppContext) { + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(json, cx) + .result() + .expect("settings parse"); + }) + }); +} + +/// A voice input whose remote service never answers, so a transcription stays in flight. +fn setup(settings_json: &str, cx: &mut TestAppContext) -> (Entity, tempfile::TempDir) { + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); + set_settings(settings_json, cx); + let tmp = tempfile::tempdir().expect("tempdir"); + let models = tmp.path().join("voice").join("models"); + let http: Arc = + FakeHttpClient::create(|_| futures::future::pending::>()); + let backend: Arc = Arc::new(Backend(Arc::default())); + cx.update(|cx| init_in(http, models, backend, cx)); + let vi = cx.update(|cx| VoiceInput::global(cx)); + cx.run_until_parked(); + (vi, tmp) +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +fn click(vi: &Entity, owner: EntityId, cx: &mut TestAppContext) { + vi.update(cx, |vi, cx| { + vi.handle( + Input::Click { + owner: Owner(owner.as_u64()), + }, + cx, + ) + }); +} + +fn view(vi: &Entity, owner: EntityId, cx: &mut TestAppContext) -> ViewState { + vi.read_with(cx, |vi, _| vi.view_state(owner)) +} + +fn host(vi: &Entity, cx: &mut TestAppContext) -> Option { + vi.read_with(cx, |vi, _| vi.remote_host()) +} + +#[gpui::test] +async fn remote_host_is_named_while_recording_and_transcribing(cx: &mut TestAppContext) { + let (vi, _tmp) = setup(REMOTE, cx); + let owner = cx.new(|_| ()).entity_id(); + assert_eq!( + host(&vi, cx).as_deref(), + Some("api.groq.com"), + "idle, remote" + ); + + click(&vi, owner, cx); + wait_until(cx, "recording", |cx| { + matches!(view(&vi, owner, cx), ViewState::Recording { .. }) + }); + assert_eq!(host(&vi, cx).as_deref(), Some("api.groq.com"), "recording"); + + // Switched to local while recording: the audio still goes to the host it was recorded for. + set_settings("{}", cx); + cx.run_until_parked(); + assert_eq!( + host(&vi, cx).as_deref(), + Some("api.groq.com"), + "the recording's own host" + ); + + click(&vi, owner, cx); + wait_until(cx, "transcribing", |cx| { + matches!(view(&vi, owner, cx), ViewState::Transcribing) + }); + assert_eq!( + host(&vi, cx).as_deref(), + Some("api.groq.com"), + "transcribing" + ); +} + +#[gpui::test] +async fn remote_host_is_none_for_a_local_recording(cx: &mut TestAppContext) { + let (vi, tmp) = setup(r#"{ "voice_input": { "model": "tiny" } }"#, cx); + // A sparse file of the pinned size counts as present, so the click records instead of asking. + let dir = tmp.path().join("voice").join("models"); + std::fs::create_dir_all(&dir).expect("dir"); + let spec = crate::models::spec(crate::models::VoiceModel::Tiny); + std::fs::File::create(dir.join(spec.file)) + .and_then(|file| file.set_len(spec.size)) + .expect("sparse model"); + + let owner = cx.new(|_| ()).entity_id(); + click(&vi, owner, cx); + wait_until(cx, "recording", |cx| { + matches!(view(&vi, owner, cx), ViewState::Recording { .. }) + }); + assert_eq!(host(&vi, cx), None, "local mode names no host"); +} diff --git a/crates/voice_input/src/remote_loopback_tests.rs b/crates/voice_input/src/remote_loopback_tests.rs new file mode 100644 index 0000000..a50c495 --- /dev/null +++ b/crates/voice_input/src/remote_loopback_tests.rs @@ -0,0 +1,416 @@ +// Story 044, task 8.4 (refine v3) — plain http to a loopback host goes direct, and an empty key +// name is no key (R8.4, R8.5, R8.8, R8.9; design.md C1.8, C1.10 `set_loopback_http`). Wired in +// the crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod remote_loopback_tests; +// +// Validation filter: `remote_loopback` (every test path is `remote_loopback_tests::…`). +// +// Contract: `VoiceInput::set_loopback_http(&mut self, http: Arc)` sets a +// proxy-less client. A plain-http (hence loopback, R8.4) remote URL is sent through it, even when +// the app's client has a proxy; an https URL is sent through the app's client; with no proxy-less +// client set, plain http behind a proxy is refused as today. `voice_input.remote.api_key_env: ""` +// sends no Authorization header; a named variable that is set is still sent as a bearer (R8.5), +// through either client. +// +// The app's client is a `FakeHttpClient` behind `HttpClientWithProxy` with a proxy URL, which is +// what `proxy()` reports for Zed's configured proxy. `ReqwestClient::without_proxy` itself is +// tested in `reqwest_client` (this crate cannot depend on it). +// Gap: reqwest's own bypass of the environment proxies (`no_proxy()`) is not exercised against a +// live proxy; that needs `set_var`, which is `unsafe` in edition 2024 and racy in a test binary. +// +// Hostile halves, the wrong collapse first: the proxy-less client is not a way around R8.4 — a +// plain-http URL to a host that only looks like loopback (`127.0.0.1.example.com`) reaches +// neither client — and it does not take https traffic away from the app's client; with no +// proxy-less client the refusal stands. Then the wrong split: `localhost` and `[::1]`, spelled +// differently from `127.0.0.1`, go direct too. Then the plain case. + +use std::{ + path::PathBuf, + sync::{ + Arc, Mutex, + atomic::{AtomicUsize, Ordering}, + }, + time::{Duration, Instant}, +}; + +use gpui::{AppContext as _, Entity, EntityId, TestAppContext, UpdateGlobal as _}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, HttpClientWithProxy, Response, Url}; +use settings::SettingsStore; + +use crate::capture::{ActiveCapture, CaptureBackend, CaptureEvent, Ended, Recording}; +use crate::state::{Input, Owner}; +use crate::{ViewState, VoiceEvent, VoiceInput, init_in}; + +const PROXY_REFUSAL: &str = + "The remote voice URL uses plain http while a proxy is configured; use https"; + +struct Mic; +struct Active; + +impl CaptureBackend for Mic { + fn start( + &self, + _device: Option, + _events: smol::channel::Sender, + ) -> anyhow::Result> { + Ok(Box::new(Active)) + } +} + +impl ActiveCapture for Active { + fn stop(self: Box) -> Recording { + let samples = (0..48_000).map(|i| 0.2 * (i as f32 * 0.05).sin()).collect(); + Recording { + samples, + sample_rate: 48_000, + ended: Ended::User, + } + } + fn cancel(self: Box) {} +} + +/// What one fake client saw. +#[derive(Clone, Default)] +struct Seen { + requests: Arc, + uris: Arc>>, + authorization: Arc>>>, +} + +impl Seen { + fn count(&self) -> usize { + self.requests.load(Ordering::SeqCst) + } + fn authorization(&self) -> Vec> { + self.authorization.lock().expect("lock").clone() + } +} + +fn answering(seen: &Seen, text: &'static str) -> Arc { + let seen = seen.clone(); + FakeHttpClient::create(move |request| { + seen.requests.fetch_add(1, Ordering::SeqCst); + seen.uris + .lock() + .expect("lock") + .push(request.uri().to_string()); + seen.authorization.lock().expect("lock").push( + request + .headers() + .get("authorization") + .and_then(|value| value.to_str().ok()) + .map(str::to_string), + ); + async move { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from(format!(r#"{{"text": "{text}"}}"#)))?) + } + }) +} + +struct Harness { + vi: Entity, + main: Seen, + loopback: Seen, + events: Arc>>, + _tmp: tempfile::TempDir, +} + +impl Harness { + fn inserts(&self, owner: EntityId) -> Vec { + self.events + .lock() + .expect("lock") + .iter() + .filter_map(|event| match event { + VoiceEvent::Insert { owner: o, text } if *o == owner => Some(text.clone()), + _ => None, + }) + .collect() + } + fn notices(&self, owner: EntityId) -> Vec { + self.events + .lock() + .expect("lock") + .iter() + .filter_map(|event| match event { + VoiceEvent::Notify { owner: o, message } if *o == owner => Some(message.clone()), + _ => None, + }) + .collect() + } + fn view(&self, owner: EntityId, cx: &mut TestAppContext) -> ViewState { + self.vi.read_with(cx, |vi, _| vi.view_state(owner)) + } + + /// Records one toggle recording for `owner` and waits until its transcript or notice arrives. + fn record(&self, owner: EntityId, cx: &mut TestAppContext) { + self.vi.update(cx, |vi, cx| { + vi.handle( + Input::Click { + owner: Owner(owner.as_u64()), + }, + cx, + ) + }); + wait_until(cx, "recording", |cx| { + matches!(self.view(owner, cx), ViewState::Recording { .. }) + }); + self.vi.update(cx, |vi, cx| { + vi.handle( + Input::Click { + owner: Owner(owner.as_u64()), + }, + cx, + ) + }); + wait_until(cx, "the transcript or a notice", |_| { + !self.inserts(owner).is_empty() || !self.notices(owner).is_empty() + }); + } +} + +fn remote_settings(url: &str, api_key_env: &str) -> String { + format!( + r#"{{ "voice_input": {{ "remote": {{ "enabled": true, "url": "{url}", "api_key_env": "{api_key_env}" }} }} }}"# + ) +} + +/// Remote mode on `url`; the app's client has a proxy; the proxy-less client is set when +/// `with_loopback`. +fn setup(url: &str, api_key_env: &str, with_loopback: bool, cx: &mut TestAppContext) -> Harness { + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); + let json = remote_settings(url, api_key_env); + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(&json, cx) + .result() + .expect("settings parse"); + }) + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let models: PathBuf = tmp.path().join("voice").join("models"); + let main = Seen::default(); + let loopback = Seen::default(); + let proxy: Url = "http://proxy.corp:3128".parse().expect("proxy url"); + let main_client: Arc = Arc::new(HttpClientWithProxy::new_url( + answering(&main, "through the main client"), + Some(proxy), + )); + assert!( + main_client.proxy().is_some(), + "precondition: the app's client has a proxy" + ); + let capture: Arc = Arc::new(Mic); + cx.update(|cx| init_in(main_client, models, capture, cx)); + let vi = cx.update(|cx| VoiceInput::global(cx)); + if with_loopback { + let direct = answering(&loopback, "sent direct"); + vi.update(cx, |vi, _| vi.set_loopback_http(direct)); + } + let events: Arc>> = Arc::default(); + let sink = events.clone(); + cx.update(|cx| { + cx.subscribe(&vi, move |_, event: &VoiceEvent, _| { + sink.lock().expect("lock").push(event.clone()) + }) + .detach() + }); + cx.run_until_parked(); + Harness { + vi, + main, + loopback, + events, + _tmp: tmp, + } +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +const UNSET_KEY: &str = "ZEO_VOICE_TEST_UNSET_044_LB"; + +// ── the wrong collapse ────────────────────────────────────────────────────── + +#[gpui::test] +async fn remote_loopback_a_lookalike_host_reaches_neither_client(cx: &mut TestAppContext) { + let h = setup("http://127.0.0.1.example.com:9/v1/t", UNSET_KEY, true, cx); + let a = cx.new(|_| ()).entity_id(); + h.record(a, cx); + assert_eq!( + (h.main.count(), h.loopback.count()), + (0, 0), + "R8.4 nothing is sent for a plain-http URL whose host is not loopback" + ); + assert!(h.inserts(a).is_empty()); + let notices = h.notices(a); + assert!( + notices.iter().any(|notice| notice.contains("https")), + "R8.4 the user is told the URL must use https: {notices:?}" + ); +} + +#[gpui::test] +async fn remote_loopback_https_still_goes_through_the_app_client(cx: &mut TestAppContext) { + let h = setup( + "https://stt.example/v1/audio/transcriptions", + UNSET_KEY, + true, + cx, + ); + let a = cx.new(|_| ()).entity_id(); + h.record(a, cx); + assert_eq!( + h.main.count(), + 1, + "https uses the app's client, with its proxy" + ); + assert_eq!( + h.loopback.count(), + 0, + "the proxy-less client is for plain http only" + ); + assert_eq!(h.inserts(a).len(), 1, "{:?}", h.notices(a)); +} + +#[gpui::test] +async fn remote_loopback_without_a_proxy_less_client_the_refusal_stands(cx: &mut TestAppContext) { + let h = setup( + "http://127.0.0.1:9/v1/audio/transcriptions", + UNSET_KEY, + false, + cx, + ); + let a = cx.new(|_| ()).entity_id(); + h.record(a, cx); + assert_eq!(h.main.count(), 0, "nothing goes through the proxied client"); + assert_eq!(h.notices(a), vec![PROXY_REFUSAL.to_string()]); + assert!(h.inserts(a).is_empty()); +} + +// ── the wrong split, then the plain case ─────────────────────────────────── + +#[gpui::test] +async fn remote_loopback_plain_http_to_127_0_0_1_goes_direct(cx: &mut TestAppContext) { + let h = setup( + "http://127.0.0.1:9/v1/audio/transcriptions", + UNSET_KEY, + true, + cx, + ); + let a = cx.new(|_| ()).entity_id(); + h.record(a, cx); + assert_eq!( + h.loopback.count(), + 1, + "R8.9 the request reaches the proxy-less client: {:?}", + h.notices(a) + ); + assert_eq!(h.main.count(), 0, "the proxied client sees nothing"); + assert_eq!( + h.inserts(a).len(), + 1, + "transcribed and inserted: {:?}", + h.notices(a) + ); + assert!( + h.inserts(a)[0].to_lowercase().contains("sent direct"), + "the transcript is the loopback service's: {:?}", + h.inserts(a) + ); +} + +#[gpui::test] +async fn remote_loopback_localhost_goes_direct(cx: &mut TestAppContext) { + let h = setup( + "http://localhost:8080/v1/audio/transcriptions", + UNSET_KEY, + true, + cx, + ); + let a = cx.new(|_| ()).entity_id(); + h.record(a, cx); + assert_eq!( + (h.loopback.count(), h.main.count()), + (1, 0), + "R8.9 localhost: {:?}", + h.notices(a) + ); +} + +#[gpui::test] +async fn remote_loopback_ipv6_loopback_goes_direct(cx: &mut TestAppContext) { + let h = setup( + "http://[::1]:8080/v1/audio/transcriptions", + UNSET_KEY, + true, + cx, + ); + let a = cx.new(|_| ()).entity_id(); + h.record(a, cx); + assert_eq!( + (h.loopback.count(), h.main.count()), + (1, 0), + "R8.9 [::1]: {:?}", + h.notices(a) + ); +} + +// ── the key name ──────────────────────────────────────────────────────────── + +#[gpui::test] +async fn remote_loopback_an_empty_key_name_sends_no_authorization(cx: &mut TestAppContext) { + let h = setup("http://127.0.0.1:9/v1/audio/transcriptions", "", true, cx); + let a = cx.new(|_| ()).entity_id(); + h.record(a, cx); + assert_eq!(h.loopback.count(), 1, "sent: {:?}", h.notices(a)); + assert_eq!( + h.loopback.authorization(), + vec![None], + "R8.8 api_key_env \"\" is no key" + ); +} + +#[gpui::test] +async fn remote_loopback_a_set_key_is_still_sent_as_a_bearer(cx: &mut TestAppContext) { + // `PATH` is set in every test environment; setting a variable here would need `unsafe`. + let Ok(value) = std::env::var("PATH") else { + return; + }; + let h = setup( + "http://127.0.0.1:9/v1/audio/transcriptions", + "PATH", + true, + cx, + ); + let a = cx.new(|_| ()).entity_id(); + h.record(a, cx); + assert_eq!(h.loopback.count(), 1, "sent: {:?}", h.notices(a)); + assert_eq!( + h.loopback.authorization(), + vec![Some(format!("Bearer {value}"))], + "R8.5 the direct request carries the key too" + ); +} diff --git a/crates/voice_input/src/remote_redirect_tests.rs b/crates/voice_input/src/remote_redirect_tests.rs new file mode 100644 index 0000000..18a6e41 --- /dev/null +++ b/crates/voice_input/src/remote_redirect_tests.rs @@ -0,0 +1,197 @@ +// Story 044, task 8.4 fix round 1 (tech review L3) — the remote request never follows a redirect, +// whichever client sends it (D22; `remote.rs` `build_request`). A redirect would carry the audio +// and the bearer key to whatever host it names, and through the proxy-less client it would leave +// the machine with no proxy in the way. +// +// `ReqwestClient::send` turns the request's `RedirectPolicy` into reqwest's per-request policy, +// which wins over the client's default (`redirect::Policy::limited(10)`). A request without the +// extension would therefore follow redirects: the test asserts the extension is present and is +// `NoFollow`, on the app client (https) and on the proxy-less client (plain http to loopback). + +use std::{ + path::PathBuf, + sync::{Arc, Mutex}, + time::{Duration, Instant}, +}; + +use gpui::{AppContext as _, Entity, EntityId, TestAppContext, UpdateGlobal as _}; +use http_client::{ + AsyncBody, FakeHttpClient, HttpClient, HttpClientWithProxy, RedirectPolicy, Response, Url, +}; +use settings::SettingsStore; + +use crate::capture::{ActiveCapture, CaptureBackend, CaptureEvent, Ended, Recording}; +use crate::state::{Input, Owner}; +use crate::{ViewState, VoiceEvent, VoiceInput, init_in}; + +struct Mic; +struct Active; + +impl CaptureBackend for Mic { + fn start( + &self, + _device: Option, + _events: smol::channel::Sender, + ) -> anyhow::Result> { + Ok(Box::new(Active)) + } +} + +impl ActiveCapture for Active { + fn stop(self: Box) -> Recording { + let samples = (0..48_000).map(|i| 0.2 * (i as f32 * 0.05).sin()).collect(); + Recording { + samples, + sample_rate: 48_000, + ended: Ended::User, + } + } + fn cancel(self: Box) {} +} + +/// The redirect policy of every request one fake client received. +type Policies = Arc>>>; + +fn recording_policies(policies: &Policies) -> Arc { + let policies = policies.clone(); + FakeHttpClient::create(move |request| { + policies + .lock() + .expect("lock") + .push(request.extensions().get::().cloned()); + async move { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from(r#"{"text": "hello"}"#.to_string()))?) + } + }) +} + +struct Sent { + main: Vec>, + loopback: Vec>, + inserts: usize, +} + +/// One toggle recording in remote mode on `url`, with an app client behind a proxy and a +/// proxy-less client set; returns what each client received. +fn record_once(url: &str, cx: &mut TestAppContext) -> Sent { + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); + let json = format!( + r#"{{ "voice_input": {{ "remote": {{ "enabled": true, "url": "{url}", "api_key_env": "" }} }} }}"# + ); + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(&json, cx) + .result() + .expect("settings parse"); + }) + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let models: PathBuf = tmp.path().join("voice").join("models"); + let main = Policies::default(); + let loopback = Policies::default(); + let proxy: Url = "http://proxy.corp:3128".parse().expect("proxy url"); + let main_client: Arc = Arc::new(HttpClientWithProxy::new_url( + recording_policies(&main), + Some(proxy), + )); + let capture: Arc = Arc::new(Mic); + cx.update(|cx| init_in(main_client, models, capture, cx)); + let vi: Entity = cx.update(|cx| VoiceInput::global(cx)); + let direct = recording_policies(&loopback); + vi.update(cx, |vi, _| vi.set_loopback_http(direct)); + let events: Arc>> = Arc::default(); + let sink = events.clone(); + cx.update(|cx| { + cx.subscribe(&vi, move |_, event: &VoiceEvent, _| { + sink.lock().expect("lock").push(event.clone()) + }) + .detach() + }); + cx.run_until_parked(); + + let owner: EntityId = cx.new(|_| ()).entity_id(); + let click = |vi: &Entity, cx: &mut TestAppContext| { + vi.update(cx, |vi, cx| { + vi.handle( + Input::Click { + owner: Owner(owner.as_u64()), + }, + cx, + ) + }) + }; + click(&vi, cx); + wait_until(cx, "recording", |cx| { + matches!( + vi.read_with(cx, |vi, _| vi.view_state(owner)), + ViewState::Recording { .. } + ) + }); + click(&vi, cx); + let answered = |events: &Mutex>| { + events.lock().expect("lock").iter().any(|event| { + matches!(event, VoiceEvent::Insert { owner: o, .. } | VoiceEvent::Notify { owner: o, .. } if *o == owner) + }) + }; + wait_until(cx, "the transcript or a notice", |_| answered(&events)); + let inserts = events + .lock() + .expect("lock") + .iter() + .filter(|event| matches!(event, VoiceEvent::Insert { owner: o, .. } if *o == owner)) + .count(); + let sent = Sent { + main: main.lock().expect("lock").clone(), + loopback: loopback.lock().expect("lock").clone(), + inserts, + }; + drop(tmp); + sent +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +#[gpui::test] +async fn remote_redirect_the_app_client_request_follows_no_redirect(cx: &mut TestAppContext) { + let sent = record_once("https://stt.example/v1/audio/transcriptions", cx); + assert_eq!(sent.inserts, 1, "transcribed through the app client"); + assert!(sent.loopback.is_empty(), "https stays on the app client"); + assert_eq!( + sent.main, + vec![Some(RedirectPolicy::NoFollow)], + "D22 a redirect would carry the key to the host it names" + ); +} + +#[gpui::test] +async fn remote_redirect_the_loopback_request_follows_no_redirect(cx: &mut TestAppContext) { + let sent = record_once("http://127.0.0.1:9/v1/audio/transcriptions", cx); + assert_eq!(sent.inserts, 1, "transcribed through the proxy-less client"); + assert!(sent.main.is_empty(), "plain http to loopback goes direct"); + assert_eq!( + sent.loopback, + vec![Some(RedirectPolicy::NoFollow)], + "D22 a loopback redirect would leave the machine with no proxy in the way" + ); +} diff --git a/crates/voice_input/src/remote_tests.rs b/crates/voice_input/src/remote_tests.rs new file mode 100644 index 0000000..237834c --- /dev/null +++ b/crates/voice_input/src/remote_tests.rs @@ -0,0 +1,143 @@ +// Story 044, task 3.4 — remote URL rules, redaction and the multipart body (R8.2, R8.4, R10.3). +// Wired in the crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod remote_tests; +// +// Validation filter: `remote` (every test path is `remote_tests::…`). +// +// Contract (design.md C1.8): `validate_url(&str) -> anyhow::Result`, `redacted(&Url) -> String`, +// `multipart(boundary: &str, wav: &[u8], model: &str, language: &Language) -> Vec`. +// +// Hostile halves: https to any host is accepted AND http to a non-loopback host is refused AND http +// to 127.0.0.0/8, ::1 and localhost is accepted; a loopback look-alike (`127.0.0.1.evil.example`, +// `localhost.evil.example`, `0.0.0.0`, `::ffff:8.8.8.8`) is refused; user-info is refused even +// with https; redaction drops user-info, query and fragment AND keeps scheme, host, port and path; +// the language field is present for a code AND absent for Auto. + +use crate::Language; +use crate::remote::{multipart, redacted, validate_url}; + +#[test] +fn remote_https_is_accepted_for_any_host() { + for raw in [ + "https://api.openai.com/v1/audio/transcriptions", + "https://api.groq.com/openai/v1/audio/transcriptions", + "https://stt.example:8443/v1/audio/transcriptions", + ] { + let url = validate_url(raw).unwrap_or_else(|e| panic!("{raw}: {e:#}")); + assert_eq!(url.scheme(), "https"); + } +} + +#[test] +fn remote_plain_http_to_a_remote_host_is_refused() { + for raw in [ + "http://api.openai.com/v1/audio/transcriptions", + "http://192.168.1.10:8080/inference", + "http://127.0.0.1.evil.example/x", + "http://localhost.evil.example/x", + "http://0.0.0.0:8080/x", + "http://[::ffff:8.8.8.8]/x", + ] { + let err = validate_url(raw).expect_err(raw); + assert!( + format!("{err:#}").contains("https"), + "R8.4 says https is required: {err:#}" + ); + } +} + +#[test] +fn remote_plain_http_to_loopback_is_accepted() { + for raw in [ + "http://127.0.0.1:8080/inference", + "http://127.10.20.30/inference", + "http://[::1]:9000/v1/audio/transcriptions", + "http://localhost:8080/inference", + ] { + validate_url(raw).unwrap_or_else(|e| panic!("{raw}: {e:#}")); + } +} + +#[test] +fn remote_user_info_is_refused_even_with_https() { + for raw in [ + "https://user:secret@api.example/v1", + "https://token@api.example/v1", + "http://u:p@127.0.0.1:8080/x", + ] { + let err = validate_url(raw).expect_err(raw); + assert!( + !format!("{err:#}").contains("secret"), + "the refusal does not echo credentials: {err:#}" + ); + } +} + +#[test] +fn remote_other_schemes_and_garbage_are_refused() { + for raw in ["ftp://api.example/x", "file:///tmp/x", "not a url", ""] { + assert!(validate_url(raw).is_err(), "{raw:?}"); + } +} + +#[test] +fn remote_redaction_keeps_only_scheme_host_port_path() { + let url = url::Url::parse( + "https://user:pw@stt.example:8443/v1/audio/transcriptions?api_key=SECRET#frag", + ) + .expect("parse"); + let shown = redacted(&url); + assert_eq!(shown, "https://stt.example:8443/v1/audio/transcriptions"); + for leaked in ["user", "pw", "SECRET", "api_key", "frag", "?", "#", "@"] { + assert!(!shown.contains(leaked), "R10.3 {leaked:?} leaked: {shown}"); + } + let plain = url::Url::parse("http://127.0.0.1/inference").expect("parse"); + assert_eq!(redacted(&plain), "http://127.0.0.1/inference"); +} + +fn part<'a>(body: &'a str, name: &str) -> Option<&'a str> { + let marker = format!("name=\"{name}\""); + let start = body.find(&marker)?; + let rest = &body[start..]; + let value_start = rest.find("\r\n\r\n")? + 4; + let value_end = rest[value_start..].find("\r\n--")?; + Some(&rest[value_start..value_start + value_end]) +} + +#[test] +fn remote_multipart_has_file_model_format_and_language() { + let wav = crate::audio_prep::wav_pcm16(&[0.0, 0.5, -0.5]); + let boundary = "zeo-boundary-7f3a"; + let body = multipart(boundary, &wav, "whisper-1", &Language::Code("pt".into())); + let text = String::from_utf8_lossy(&body).into_owned(); + + assert!(text.starts_with(&format!("--{boundary}\r\n")), "{text:.80}"); + assert!( + text.trim_end().ends_with(&format!("--{boundary}--")), + "closing delimiter" + ); + assert!( + text.contains("name=\"file\"; filename=\"audio.wav\""), + "{text:.200}" + ); + assert!(text.contains("Content-Type: audio/wav")); + assert_eq!(part(&text, "model"), Some("whisper-1")); + assert_eq!(part(&text, "response_format"), Some("json")); + assert_eq!(part(&text, "language"), Some("pt")); + let at = body.windows(wav.len()).position(|w| w == wav.as_slice()); + assert!(at.is_some(), "the WAV bytes are carried verbatim"); +} + +#[test] +fn remote_multipart_omits_language_for_auto() { + let wav = crate::audio_prep::wav_pcm16(&[0.0; 4]); + let body = multipart("b", &wav, "whisper-1", &Language::Auto); + let text = String::from_utf8_lossy(&body); + assert!( + !text.contains("name=\"language\""), + "R7.4 Auto lets the service detect" + ); + assert!(text.contains("name=\"model\"")); +} diff --git a/crates/voice_input/src/settings.rs b/crates/voice_input/src/settings.rs new file mode 100644 index 0000000..4c08003 --- /dev/null +++ b/crates/voice_input/src/settings.rs @@ -0,0 +1,84 @@ +use settings::{RegisterSetting, Settings, SettingsContent, VoiceModelContent}; + +use crate::VoiceModel; + +/// The `voice_input` settings section. +#[derive(Clone, Debug, PartialEq, Eq, RegisterSetting)] +pub struct VoiceInputSettings { + /// Whether the composer shows the microphone button and binds the voice shortcut. + pub enabled: bool, + /// The local Whisper model. + pub model: VoiceModel, + /// The spoken language, or `Auto` to let the model detect it. + pub language: Language, + /// The optional remote transcription service. + pub remote: RemoteSettings, +} + +/// The `voice_input.remote` settings subsection. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RemoteSettings { + /// Whether audio goes to the remote service instead of the local model. Off by default. + pub enabled: bool, + /// The transcription endpoint; `None` when unset. + pub url: Option, + /// The model name sent to the remote service. + pub model: String, + /// The environment variable holding the API key (the key itself is never in settings). + pub api_key_env: String, +} + +/// The language Whisper transcribes in. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Language { + /// The model detects the spoken language. + Auto, + /// A two-letter lowercase language code, such as `en` or `pt`. + Code(String), +} + +impl Language { + /// Parses `"auto"` or two ASCII lowercase letters. Anything else falls back to `Auto` + /// with one warning; the refused value is not logged. + pub fn parse(raw: &str) -> Self { + if raw == "auto" { + return Self::Auto; + } + if raw.len() == 2 && raw.bytes().all(|byte| byte.is_ascii_lowercase()) { + return Self::Code(raw.to_owned()); + } + log::warn!("voice_input event=setting_invalid key=language"); + Self::Auto + } +} + +// The defaults below repeat `assets/settings/default.json`, so an embedder whose defaults lack +// the section still gets the design's values. +const DEFAULT_ENABLED: bool = true; +const DEFAULT_MODEL: VoiceModelContent = VoiceModelContent::Base; +const DEFAULT_LANGUAGE: &str = "auto"; +const DEFAULT_REMOTE_ENABLED: bool = false; +const DEFAULT_REMOTE_MODEL: &str = "whisper-1"; +const DEFAULT_REMOTE_API_KEY_ENV: &str = "ZEO_VOICE_API_KEY"; + +impl Settings for VoiceInputSettings { + fn from_settings(content: &SettingsContent) -> Self { + let section = content.voice_input.clone().unwrap_or_default(); + let remote = section.remote.unwrap_or_default(); + Self { + enabled: section.enabled.unwrap_or(DEFAULT_ENABLED), + model: section.model.unwrap_or(DEFAULT_MODEL).into(), + language: Language::parse(section.language.as_deref().unwrap_or(DEFAULT_LANGUAGE)), + remote: RemoteSettings { + enabled: remote.enabled.unwrap_or(DEFAULT_REMOTE_ENABLED), + url: remote.url, + model: remote + .model + .unwrap_or_else(|| DEFAULT_REMOTE_MODEL.to_owned()), + api_key_env: remote + .api_key_env + .unwrap_or_else(|| DEFAULT_REMOTE_API_KEY_ENV.to_owned()), + }, + } + } +} diff --git a/crates/voice_input/src/settings_tests.rs b/crates/voice_input/src/settings_tests.rs new file mode 100644 index 0000000..38bfa41 --- /dev/null +++ b/crates/voice_input/src/settings_tests.rs @@ -0,0 +1,132 @@ +// Story 044, task 1.2 — the `voice_input` settings section (R1.2, R7.2, R7.4, R8.1). Wired in the +// crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod settings_tests; +// +// Validation filter: `settings` (every test path is `settings_tests::…`). +// +// Contract (design.md C1.1, Data Models). Assumed shapes — adjust the `use` only: +// - `VoiceInputSettings { enabled, model, language, remote }` implements `settings::Settings` +// (`#[derive(RegisterSetting)]`, so `SettingsStore::test` knows it). +// - `RemoteSettings { enabled, url: Option, model: String, api_key_env: String }`. +// - `Language::{Auto, Code(String)}` with `Language::parse(&str) -> Language`. +// - `VoiceModel::{Tiny, Base, Small}`; `PartialEq + Debug` on `VoiceModel` and `Language`. +// +// Hostile halves: a value in the user's settings.json is READ (a field missing from +// `flattened_deserialize!` or `vscode_import.rs` is silently ignored — 0052's lesson), AND an absent +// value falls back to the default; the language parser accepts two lowercase letters AND refuses +// near misses (uppercase, three letters, digits, empty), which fall back to Auto. + +use gpui::{TestAppContext, UpdateGlobal as _}; +use settings::{Settings as _, SettingsStore}; + +use crate::{Language, VoiceInputSettings, VoiceModel}; + +fn init(cx: &mut TestAppContext) { + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); +} + +fn set_user_settings(json: &str, cx: &mut TestAppContext) { + cx.update(|cx| { + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(json, cx) + .result() + .expect("user settings parse"); + }) + }); +} + +fn current(cx: &mut TestAppContext) -> VoiceInputSettings { + cx.update(|cx| VoiceInputSettings::get_global(cx).clone()) +} + +#[gpui::test] +fn settings_defaults_match_the_design(cx: &mut TestAppContext) { + init(cx); + let s = current(cx); + assert!(s.enabled, "voice_input.enabled defaults to true"); + assert_eq!(s.model, VoiceModel::Base, "the default model is base"); + assert_eq!(s.language, Language::Auto); + assert!( + !s.remote.enabled, + "R8.1 the remote service is off by default" + ); + assert_eq!(s.remote.url, None); + assert_eq!(s.remote.model, "whisper-1"); + assert_eq!(s.remote.api_key_env, "ZEO_VOICE_API_KEY"); +} + +#[gpui::test] +fn settings_user_values_are_read_not_ignored(cx: &mut TestAppContext) { + init(cx); + set_user_settings( + r#"{ "voice_input": { + "enabled": false, + "model": "small", + "language": "pt", + "remote": { "enabled": true, "url": "https://stt.example/v1/audio/transcriptions", + "model": "large-v3", "api_key_env": "MY_STT_KEY" } + } }"#, + cx, + ); + let s = current(cx); + assert!( + !s.enabled, + "R1.2 a user `enabled: false` reaches the setting" + ); + assert_eq!(s.model, VoiceModel::Small); + assert_eq!(s.language, Language::Code("pt".into()), "R7.2"); + assert!(s.remote.enabled); + assert_eq!( + s.remote.url.as_deref(), + Some("https://stt.example/v1/audio/transcriptions") + ); + assert_eq!(s.remote.model, "large-v3"); + assert_eq!(s.remote.api_key_env, "MY_STT_KEY"); +} + +#[gpui::test] +fn settings_partial_user_section_keeps_other_defaults(cx: &mut TestAppContext) { + init(cx); + set_user_settings( + r#"{ "voice_input": { "remote": { "enabled": true } } }"#, + cx, + ); + let s = current(cx); + assert!(s.enabled && s.remote.enabled); + assert_eq!(s.model, VoiceModel::Base); + assert_eq!(s.remote.model, "whisper-1"); + assert_eq!(s.remote.api_key_env, "ZEO_VOICE_API_KEY"); +} + +#[gpui::test] +fn settings_tiny_model_parses(cx: &mut TestAppContext) { + init(cx); + set_user_settings(r#"{ "voice_input": { "model": "tiny" } }"#, cx); + assert_eq!(current(cx).model, VoiceModel::Tiny); +} + +#[test] +fn settings_language_auto_and_two_lowercase_letters() { + assert_eq!(Language::parse("auto"), Language::Auto, "R7.4"); + assert_eq!(Language::parse("en"), Language::Code("en".into()), "R7.2"); + assert_eq!(Language::parse("pt"), Language::Code("pt".into())); +} + +#[test] +fn settings_language_near_misses_fall_back_to_auto() { + for raw in [ + "EN", "Pt", "por", "p", "", " en", "en ", "e1", "12", "pt-BR", "AUTO", "ée", + ] { + assert_eq!( + Language::parse(raw), + Language::Auto, + "{raw:?} is not a two-letter code" + ); + } +} diff --git a/crates/voice_input/src/state.rs b/crates/voice_input/src/state.rs new file mode 100644 index 0000000..9ab0e6a --- /dev/null +++ b/crates/voice_input/src/state.rs @@ -0,0 +1,401 @@ +//! The voice-input state machine: what every input means, with no I/O and no gpui. + +use std::time::{Duration, Instant}; + +/// The composer a request belongs to; the entity builds it from `EntityId::as_u64()`. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct Owner(pub u64); + +/// How a recording stops: `Pending` until a shortcut press is resolved into a tap or a hold. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Mode { + Pending(Instant), + Toggle, + Hold, +} + +/// What started the recording while the device is being opened. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Via { + Click, + Shortcut { pressed_at: Instant, released: bool }, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Phase { + Idle, + AwaitingConsent { + owner: Owner, + }, + Downloading { + owner: Owner, + bytes: u64, + }, + Starting { + owner: Owner, + via: Via, + }, + Recording { + owner: Owner, + mode: Mode, + started: Instant, + }, + Transcribing { + owner: Owner, + }, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Input { + Click { owner: Owner }, + ShortcutDown { owner: Owner }, + ShortcutUp { owner: Owner }, + Cancel { owner: Owner }, + WindowDeactivated { owner: Owner }, + CaptureStarted, + CaptureFailed(String), + CaptureLimit, + CaptureDeviceError(String), + Transcribed { text: String }, + TranscribeFailed(String), + ConsentGiven, + ConsentDeclined, + DownloadProgress(u64), + DownloadDone, + DownloadFailed(String), + DownloadCancel, + AdoptDownload { owner: Owner }, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Effect { + StartCapture, + StopCapture, + CancelCapture, + AskConsent, + StartDownload, + CancelDownload, + Transcribe { mode: Mode }, + Insert { owner: Owner, text: String }, + Notify { owner: Owner, message: String }, + Busy { owner: Owner }, + Render, +} + +/// What the entity knows about the requesting composer and the settings at input time. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Ctx { + pub enabled: bool, + pub remote: bool, + /// `resolve_model(..).is_some()` for the configured local model. + pub model_present: bool, + pub composer_read_only: bool, +} + +#[derive(Debug)] +pub struct Machine { + phase: Phase, + shortcut_down: bool, +} + +impl Default for Machine { + fn default() -> Self { + Self { + phase: Phase::Idle, + shortcut_down: false, + } + } +} + +/// A shortcut released within this long of its press is a tap (toggle); longer is a hold (R2.2, R3.1). +pub const TAP_WINDOW: Duration = Duration::from_millis(300); + +const NO_MICROPHONE: &str = "No microphone available"; +const MICROPHONE_STOPPED: &str = "The microphone stopped"; +const LIMIT_REACHED: &str = "Recording stopped at the 2-minute limit"; +const NO_SPEECH: &str = "No speech was recognised"; + +impl Machine { + pub fn phase(&self) -> &Phase { + &self.phase + } + + /// Applies one input and returns what the caller must do. Inputs that do not apply to the + /// current phase return no effect. A `Render` is appended whenever the phase changed. + pub fn on(&mut self, input: Input, now: Instant, ctx: &Ctx) -> Vec { + let before = self.phase.clone(); + let mut effects = self.apply(input, now, ctx); + if self.phase != before && !effects.contains(&Effect::Render) { + effects.push(Effect::Render); + } + effects + } + + fn apply(&mut self, input: Input, now: Instant, ctx: &Ctx) -> Vec { + match input { + Input::Click { owner } => self.start_input(owner, Via::Click, ctx), + Input::ShortcutDown { owner } => { + // X11 delivers auto-repeat as fresh key-downs with `is_held: false`, so the + // machine, not the event, knows whether the key is already down (R3.2). + if self.shortcut_down { + return Vec::new(); + } + self.shortcut_down = true; + let via = Via::Shortcut { + pressed_at: now, + released: false, + }; + self.start_input(owner, via, ctx) + } + Input::ShortcutUp { .. } => { + self.shortcut_down = false; + self.shortcut_released(now) + } + Input::Cancel { owner } => match self.phase { + Phase::Recording { owner: current, .. } if current == owner => { + self.phase = Phase::Idle; + vec![Effect::CancelCapture] + } + _ => Vec::new(), + }, + Input::WindowDeactivated { owner } => { + // The key-up of a shortcut held across the focus loss goes to another window, + // so the next press here is a fresh one, not a repeat (R3.2, R3.3). + self.shortcut_down = false; + self.window_deactivated(owner, now) + } + Input::CaptureStarted => match self.phase { + Phase::Starting { owner, via } => { + let mode = match via { + Via::Shortcut { + pressed_at, + released: false, + } => Mode::Pending(pressed_at), + // A tap released before the device opened is still a tap (R2.2). + Via::Shortcut { released: true, .. } | Via::Click => Mode::Toggle, + }; + self.phase = Phase::Recording { + owner, + mode, + started: now, + }; + Vec::new() + } + _ => Vec::new(), + }, + Input::CaptureFailed(reason) => match self.phase { + Phase::Starting { owner, .. } => { + self.phase = Phase::Idle; + vec![notify(owner, format!("{NO_MICROPHONE}: {reason}"))] + } + _ => Vec::new(), + }, + Input::CaptureLimit => self.stop_with_notice(now, LIMIT_REACHED.to_string()), + Input::CaptureDeviceError(reason) => { + self.stop_with_notice(now, format!("{MICROPHONE_STOPPED}: {reason}")) + } + Input::Transcribed { text } => match self.phase { + Phase::Transcribing { owner } => { + self.phase = Phase::Idle; + let text = crate::transcript::clean(&text); + if text.is_empty() { + vec![notify(owner, NO_SPEECH.to_string())] + } else { + vec![Effect::Insert { owner, text }] + } + } + _ => Vec::new(), + }, + Input::TranscribeFailed(message) => match self.phase { + Phase::Transcribing { owner } => { + self.phase = Phase::Idle; + vec![notify(owner, message)] + } + _ => Vec::new(), + }, + Input::ConsentGiven => match self.phase { + Phase::AwaitingConsent { owner } => { + self.phase = Phase::Downloading { owner, bytes: 0 }; + vec![Effect::StartDownload] + } + _ => Vec::new(), + }, + Input::ConsentDeclined => match self.phase { + Phase::AwaitingConsent { .. } => { + self.phase = Phase::Idle; + Vec::new() + } + _ => Vec::new(), + }, + Input::DownloadProgress(downloaded) => match &mut self.phase { + Phase::Downloading { bytes, .. } => { + *bytes = downloaded; + vec![Effect::Render] + } + _ => Vec::new(), + }, + Input::DownloadDone => match self.phase { + // Recording is not started here: the moment of consent is not the moment the + // user is ready to speak. + Phase::Downloading { .. } => { + self.phase = Phase::Idle; + vec![Effect::Render] + } + _ => Vec::new(), + }, + Input::DownloadFailed(message) => match self.phase { + Phase::Downloading { owner, .. } => { + self.phase = Phase::Idle; + vec![notify(owner, message)] + } + _ => Vec::new(), + }, + Input::DownloadCancel => match self.phase { + Phase::Downloading { .. } => { + self.phase = Phase::Idle; + vec![Effect::CancelDownload] + } + _ => Vec::new(), + }, + Input::AdoptDownload { owner } => match &mut self.phase { + // The download outlives the composer that confirmed it: the next composer to + // start voice input takes it over, progress and all (R6.14, R6.15). + Phase::Downloading { owner: current, .. } => { + *current = owner; + vec![Effect::Render] + } + _ => Vec::new(), + }, + } + } + + /// A click or a fresh shortcut press: starts from `Idle`, stops a running toggle, and is + /// refused as busy for any other owner (R5.1). + fn start_input(&mut self, owner: Owner, via: Via, ctx: &Ctx) -> Vec { + // A composer can turn read-only, or the feature be disabled, while its own recording + // runs; the guard below refuses starts only, so the owner can still stop (R1.3, R2.3). + if let Phase::Recording { + owner: current, + mode, + .. + } = self.phase + && current == owner + { + // Only a click reaches `Pending` here (a shortcut press there is a repeat), and it + // stops like a toggle stop (R2.3). + let mode = match mode { + Mode::Hold => Mode::Hold, + Mode::Pending(_) | Mode::Toggle => Mode::Toggle, + }; + return self.stop(owner, mode); + } + if !ctx.enabled || ctx.composer_read_only { + return Vec::new(); + } + match self.phase { + Phase::Idle => { + if !ctx.remote && !ctx.model_present { + self.phase = Phase::AwaitingConsent { owner }; + vec![Effect::AskConsent] + } else { + self.phase = Phase::Starting { owner, via }; + vec![Effect::StartCapture] + } + } + Phase::AwaitingConsent { owner: current } + | Phase::Downloading { owner: current, .. } + | Phase::Starting { owner: current, .. } + | Phase::Transcribing { owner: current } + if current != owner => + { + vec![Effect::Busy { owner }] + } + // The owner's own recording was stopped above. + Phase::Recording { .. } => vec![Effect::Busy { owner }], + // The owner's own start while consenting, downloading, opening the device or + // transcribing changes nothing (R5.2). + Phase::AwaitingConsent { .. } + | Phase::Downloading { .. } + | Phase::Starting { .. } + | Phase::Transcribing { .. } => Vec::new(), + } + } + + fn shortcut_released(&mut self, now: Instant) -> Vec { + match &mut self.phase { + Phase::Starting { + via: Via::Shortcut { released, .. }, + .. + } => { + *released = true; + Vec::new() + } + Phase::Recording { + owner, + mode: mode @ Mode::Pending(_), + .. + } => match settled(*mode, now) { + Mode::Hold => { + let owner = *owner; + self.stop(owner, Mode::Hold) + } + settled_mode => { + *mode = settled_mode; + Vec::new() + } + }, + Phase::Recording { + owner, + mode: Mode::Hold, + .. + } => { + let owner = *owner; + self.stop(owner, Mode::Hold) + } + _ => Vec::new(), + } + } + + fn window_deactivated(&mut self, owner: Owner, now: Instant) -> Vec { + match self.phase { + Phase::Recording { + owner: current, + mode: mode @ (Mode::Pending(_) | Mode::Hold), + .. + } if current == owner => self.stop(current, settled(mode, now)), + _ => Vec::new(), + } + } + + fn stop(&mut self, owner: Owner, mode: Mode) -> Vec { + self.phase = Phase::Transcribing { owner }; + vec![Effect::StopCapture, Effect::Transcribe { mode }] + } + + fn stop_with_notice(&mut self, now: Instant, message: String) -> Vec { + match self.phase { + Phase::Recording { owner, mode, .. } => { + let mut effects = self.stop(owner, settled(mode, now)); + effects.push(notify(owner, message)); + effects + } + _ => Vec::new(), + } + } +} + +/// Resolves a `Pending` press into the mode it has become by `now`: within the tap window it is +/// a toggle, past it a hold. +fn settled(mode: Mode, now: Instant) -> Mode { + match mode { + Mode::Pending(pressed_at) if now.saturating_duration_since(pressed_at) <= TAP_WINDOW => { + Mode::Toggle + } + Mode::Pending(_) => Mode::Hold, + mode => mode, + } +} + +fn notify(owner: Owner, message: String) -> Effect { + Effect::Notify { owner, message } +} diff --git a/crates/voice_input/src/state_release_tests.rs b/crates/voice_input/src/state_release_tests.rs new file mode 100644 index 0000000..2f5e8e0 --- /dev/null +++ b/crates/voice_input/src/state_release_tests.rs @@ -0,0 +1,105 @@ +// Story 044, task 5.3 fix round 1 — a release seen twice. The thread view sends `ShortcutUp` both +// for the key-up of the shortcut's key and for its modifiers being let go of (macOS delivers no +// key-up while Cmd is held), so one physical release can arrive as two inputs. The second must +// change nothing, whatever phase the first left behind (R2.2, R2.3, R3.1, R3.2). + +use std::time::{Duration, Instant}; + +use crate::state::{Ctx, Effect, Input, Machine, Mode, Owner, Phase}; + +const A: Owner = Owner(1); + +fn ready() -> Ctx { + Ctx { + enabled: true, + remote: false, + model_present: true, + composer_read_only: false, + } +} + +fn ms(t0: Instant, n: u64) -> Instant { + t0 + Duration::from_millis(n) +} + +fn stops(effects: &[Effect]) -> bool { + effects + .iter() + .any(|effect| matches!(effect, Effect::StopCapture)) +} + +#[test] +fn state_release_seen_twice_after_a_hold_changes_nothing() { + let t0 = Instant::now(); + let mut machine = Machine::default(); + machine.on(Input::ShortcutDown { owner: A }, t0, &ready()); + machine.on(Input::CaptureStarted, t0, &ready()); + let first = machine.on(Input::ShortcutUp { owner: A }, ms(t0, 800), &ready()); + assert!(stops(&first), "the first release ends the hold: {first:?}"); + let second = machine.on(Input::ShortcutUp { owner: A }, ms(t0, 810), &ready()); + assert!( + second.is_empty(), + "the second release is a no-op: {second:?}" + ); + assert!( + matches!(machine.phase(), Phase::Transcribing { owner } if *owner == A), + "{:?}", + machine.phase() + ); +} + +#[test] +fn state_release_seen_twice_after_a_tap_keeps_the_toggle() { + let t0 = Instant::now(); + let mut machine = Machine::default(); + machine.on(Input::ShortcutDown { owner: A }, t0, &ready()); + machine.on(Input::CaptureStarted, t0, &ready()); + machine.on(Input::ShortcutUp { owner: A }, ms(t0, 100), &ready()); + // Past the tap window: a second release read as a hold's would stop the toggle. + let second = machine.on(Input::ShortcutUp { owner: A }, ms(t0, 900), &ready()); + assert!( + second.is_empty(), + "the second release is a no-op: {second:?}" + ); + assert!( + matches!( + machine.phase(), + Phase::Recording { + mode: Mode::Toggle, + .. + } + ), + "{:?}", + machine.phase() + ); + let press = machine.on(Input::ShortcutDown { owner: A }, ms(t0, 3000), &ready()); + assert!( + stops(&press), + "the next press is a fresh one and stops: {press:?}" + ); +} + +#[test] +fn state_release_seen_twice_before_the_device_opened_is_still_a_tap() { + let t0 = Instant::now(); + let mut machine = Machine::default(); + machine.on(Input::ShortcutDown { owner: A }, t0, &ready()); + machine.on(Input::ShortcutUp { owner: A }, ms(t0, 50), &ready()); + let second = machine.on(Input::ShortcutUp { owner: A }, ms(t0, 60), &ready()); + assert!( + second.is_empty(), + "the second release is a no-op: {second:?}" + ); + machine.on(Input::CaptureStarted, ms(t0, 400), &ready()); + assert!( + matches!( + machine.phase(), + Phase::Recording { + mode: Mode::Toggle, + .. + } + ), + "{:?}", + machine.phase() + ); +} diff --git a/crates/voice_input/src/state_stop_tests.rs b/crates/voice_input/src/state_stop_tests.rs new file mode 100644 index 0000000..1bfef03 --- /dev/null +++ b/crates/voice_input/src/state_stop_tests.rs @@ -0,0 +1,217 @@ +// Story 044, task 5.3 — a stop is never refused (R1.3, R2.3, R3.1). Wired in the crate root +// `src/voice_input.rs` as +// +// #[cfg(test)] +// mod state_stop_tests; +// +// Validation filter: `state` (every test path is `state_stop_tests::…`). +// +// A composer can turn read-only, or the feature be disabled, while its own recording runs. The +// read-only/disabled guard refuses starts only: a click or a fresh press from the recording owner, +// and the release of a held shortcut, still stop and transcribe. A press after a focus loss is a +// fresh press: the release of the held shortcut went to another window. +// +// Hostile halves: the owner's stop goes through while read-only AND a start from idle is still +// refused; the owner's stop goes through while disabled AND another owner's input stays busy, not +// a stop. + +use std::time::{Duration, Instant}; + +use crate::state::{Ctx, Effect, Input, Machine, Mode, Owner, Phase}; + +const A: Owner = Owner(1); +const B: Owner = Owner(2); + +fn ready() -> Ctx { + Ctx { + enabled: true, + remote: false, + model_present: true, + composer_read_only: false, + } +} + +fn read_only() -> Ctx { + Ctx { + composer_read_only: true, + ..ready() + } +} + +fn disabled() -> Ctx { + Ctx { + enabled: false, + ..ready() + } +} + +fn ms(t0: Instant, n: u64) -> Instant { + t0 + Duration::from_millis(n) +} + +fn stops(effects: &[Effect]) -> bool { + effects.iter().any(|e| matches!(e, Effect::StopCapture)) + && effects + .iter() + .any(|e| matches!(e, Effect::Transcribe { .. })) +} + +fn starts(effects: &[Effect]) -> bool { + effects + .iter() + .any(|e| matches!(e, Effect::StartCapture | Effect::AskConsent)) +} + +/// A toggle recording for `A`, started while the composer was writable. +fn toggle_recording(t0: Instant) -> Machine { + let mut m = Machine::default(); + m.on(Input::Click { owner: A }, t0, &ready()); + m.on(Input::CaptureStarted, t0, &ready()); + assert!( + matches!( + m.phase(), + Phase::Recording { + mode: Mode::Toggle, + .. + } + ), + "{:?}", + m.phase() + ); + m +} + +/// A held shortcut for `A` (still `Pending`), started while the composer was writable. +fn held_recording(t0: Instant) -> Machine { + let mut m = Machine::default(); + m.on(Input::ShortcutDown { owner: A }, t0, &ready()); + m.on(Input::CaptureStarted, t0, &ready()); + assert!( + matches!( + m.phase(), + Phase::Recording { + mode: Mode::Pending(_), + .. + } + ), + "{:?}", + m.phase() + ); + m +} + +#[test] +fn state_click_stops_a_recording_whose_composer_turned_read_only() { + let t0 = Instant::now(); + let mut m = toggle_recording(t0); + let e = m.on(Input::Click { owner: A }, ms(t0, 2000), &read_only()); + assert!( + stops(&e), + "the owner's click stops even when read-only: {e:?}" + ); + assert!( + matches!(m.phase(), Phase::Transcribing { owner } if *owner == A), + "{:?}", + m.phase() + ); +} + +#[test] +fn state_fresh_press_stops_a_toggle_whose_composer_turned_read_only() { + let t0 = Instant::now(); + let mut m = Machine::default(); + m.on(Input::ShortcutDown { owner: A }, t0, &ready()); + m.on(Input::CaptureStarted, t0, &ready()); + m.on(Input::ShortcutUp { owner: A }, ms(t0, 100), &ready()); + let e = m.on(Input::ShortcutDown { owner: A }, ms(t0, 3000), &read_only()); + assert!( + stops(&e), + "a fresh press stops the toggle even when read-only: {e:?}" + ); +} + +#[test] +fn state_release_stops_a_hold_whose_composer_turned_read_only() { + let t0 = Instant::now(); + let mut m = held_recording(t0); + let e = m.on(Input::ShortcutUp { owner: A }, ms(t0, 1500), &read_only()); + assert!( + stops(&e), + "releasing a hold stops even when read-only: {e:?}" + ); +} + +#[test] +fn state_click_stops_a_recording_after_the_feature_was_disabled() { + let t0 = Instant::now(); + let mut m = toggle_recording(t0); + let e = m.on(Input::Click { owner: A }, ms(t0, 2000), &disabled()); + assert!( + stops(&e), + "the owner's click stops even when disabled: {e:?}" + ); +} + +#[test] +fn state_read_only_or_disabled_still_refuses_a_start() { + let t0 = Instant::now(); + for ctx in [read_only(), disabled()] { + let mut m = Machine::default(); + let e = m.on(Input::Click { owner: A }, t0, &ctx); + assert!(!starts(&e), "{ctx:?}: {e:?}"); + let e = m.on(Input::ShortcutDown { owner: A }, t0, &ctx); + assert!(!starts(&e), "{ctx:?}: {e:?}"); + assert!(matches!(m.phase(), Phase::Idle), "{:?}", m.phase()); + + // After a stop, the next start is refused again. + let mut m = toggle_recording(t0); + m.on(Input::Click { owner: A }, ms(t0, 1000), &ctx); + m.on( + Input::Transcribed { + text: "hello".into(), + }, + ms(t0, 1100), + &ctx, + ); + let e = m.on(Input::Click { owner: A }, ms(t0, 1200), &ctx); + assert!(!starts(&e), "{ctx:?}: a start after the stop: {e:?}"); + } +} + +#[test] +fn state_another_owner_read_only_is_not_a_stop() { + let t0 = Instant::now(); + let mut m = toggle_recording(t0); + for ctx in [read_only(), disabled()] { + let e = m.on(Input::Click { owner: B }, ms(t0, 1000), &ctx); + assert!(!stops(&e), "{ctx:?}: B cannot stop A's recording: {e:?}"); + } + assert!( + matches!(m.phase(), Phase::Recording { owner, .. } if *owner == A), + "{:?}", + m.phase() + ); +} + +// The release of a shortcut held while the window lost focus goes to another application, so +// the next press in Zeo is a fresh press, not a repeat of one still held (R3.2, R3.3). +#[test] +fn state_press_after_focus_loss_is_not_taken_for_a_repeat() { + let t0 = Instant::now(); + let mut m = held_recording(t0); + let e = m.on( + Input::WindowDeactivated { owner: A }, + ms(t0, 1000), + &ready(), + ); + assert!(stops(&e), "focus loss stops the hold: {e:?}"); + m.on( + Input::Transcribed { + text: "hello".into(), + }, + ms(t0, 1100), + &ready(), + ); + let e = m.on(Input::ShortcutDown { owner: A }, ms(t0, 5000), &ready()); + assert!(starts(&e), "the next press starts a new recording: {e:?}"); +} diff --git a/crates/voice_input/src/state_tests.rs b/crates/voice_input/src/state_tests.rs new file mode 100644 index 0000000..c6ee814 --- /dev/null +++ b/crates/voice_input/src/state_tests.rs @@ -0,0 +1,671 @@ +// Story 044, task 2.3 — the pure state machine (R2.1, R2.2, R2.3, R2.5, R2.6, R3.1, R3.2, R3.3, +// R4.6, R5.1, R5.2, R6.1, R6.7, R9.1, R9.2). Wired in the crate root `src/voice_input.rs` as +// +// #[cfg(test)] +// mod state_tests; +// +// Validation filter: `state` (every test path is `state_tests::…`). +// +// Contract (design.md C1.9). Assumed shapes — adjust the `use` only: +// - `Machine::default()`, `Machine::on(&mut self, Input, Instant, &Ctx) -> Vec`, +// `Machine::phase(&self) -> &Phase` (read-only accessor for tests and `view_state`). +// - `Owner` is a `Copy + Eq + Debug` newtype `Owner(pub u64)` (the machine has no gpui; the +// entity maps `EntityId::as_u64()` onto it). +// - `Mode { Pending(Instant), Toggle, Hold }`, used by `Phase::Recording { mode, .. }` and +// `Effect::Transcribe { mode }`. +// - `Ctx { enabled, remote, model_present, composer_read_only }`, all `bool`. +// +// Hostile halves: the 300 ms rule at 299/300/301 ms; key repeats ignored AND a fresh press after +// the release accepted; a non-owner's start refused as busy AND the owner's own click stopping; +// a tap released before the capture has even started still toggles; Cancel from another owner +// does not discard this owner's audio; focus loss stops a hold AND leaves a toggle running. + +use std::time::{Duration, Instant}; + +use crate::state::{Ctx, Effect, Input, Machine, Mode, Owner, Phase}; + +const A: Owner = Owner(1); +const B: Owner = Owner(2); + +fn local_ready() -> Ctx { + Ctx { + enabled: true, + remote: false, + model_present: true, + composer_read_only: false, + } +} + +fn ms(t0: Instant, n: u64) -> Instant { + t0 + Duration::from_millis(n) +} + +fn has(effects: &[Effect], pred: impl Fn(&Effect) -> bool) -> bool { + effects.iter().any(pred) +} + +fn starts_capture(e: &[Effect]) -> bool { + has(e, |e| matches!(e, Effect::StartCapture)) +} +fn stops_capture(e: &[Effect]) -> bool { + has(e, |e| matches!(e, Effect::StopCapture)) +} +fn transcribes(e: &[Effect]) -> bool { + has(e, |e| matches!(e, Effect::Transcribe { .. })) +} +fn notify_text(e: &[Effect], owner: Owner) -> Option { + e.iter().find_map(|e| match e { + Effect::Notify { owner: o, message } if *o == owner => Some(message.to_string()), + _ => None, + }) +} + +fn is_idle(m: &Machine) -> bool { + matches!(m.phase(), Phase::Idle) +} +fn recording_mode(m: &Machine) -> Option { + match m.phase() { + Phase::Recording { mode, .. } => Some(match mode { + Mode::Pending(_) => "pending".into(), + Mode::Toggle => "toggle".into(), + Mode::Hold => "hold".into(), + }), + _ => None, + } +} + +/// Shortcut pressed at t0 and the capture reported started at t0; returns the machine in Pending. +fn pressed(t0: Instant, ctx: &Ctx) -> Machine { + let mut m = Machine::default(); + let e = m.on(Input::ShortcutDown { owner: A }, t0, ctx); + assert!( + starts_capture(&e), + "a press while idle starts the capture: {e:?}" + ); + m.on(Input::CaptureStarted, t0, ctx); + assert_eq!( + recording_mode(&m).as_deref(), + Some("pending"), + "{:?}", + m.phase() + ); + m +} + +/// Clicked at t0, capture started: Toggle. +fn clicked(t0: Instant, ctx: &Ctx) -> Machine { + let mut m = Machine::default(); + let e = m.on(Input::Click { owner: A }, t0, ctx); + assert!( + starts_capture(&e), + "a click while idle starts the capture: {e:?}" + ); + m.on(Input::CaptureStarted, t0, ctx); + assert_eq!( + recording_mode(&m).as_deref(), + Some("toggle"), + "{:?}", + m.phase() + ); + m +} + +// ── R2.2 / R3.1: the 300 ms boundary, hostile first ───────────────────────── + +#[test] +fn state_release_at_301_ms_is_hold_and_stops() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = pressed(t0, &ctx); + let e = m.on(Input::ShortcutUp { owner: A }, ms(t0, 301), &ctx); + assert!( + stops_capture(&e) && transcribes(&e), + "301 ms is a hold: stop + transcribe: {e:?}" + ); + assert!( + has(&e, |e| matches!(e, Effect::Transcribe { mode: Mode::Hold })), + "the transcription is tagged hold: {e:?}" + ); + assert!( + matches!(m.phase(), Phase::Transcribing { owner } if *owner == A), + "{:?}", + m.phase() + ); +} + +#[test] +fn state_release_at_300_ms_is_still_toggle() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = pressed(t0, &ctx); + let e = m.on(Input::ShortcutUp { owner: A }, ms(t0, 300), &ctx); + assert!( + !stops_capture(&e), + "300 ms is the inclusive toggle edge: keep recording: {e:?}" + ); + assert_eq!(recording_mode(&m).as_deref(), Some("toggle")); +} + +#[test] +fn state_release_at_299_ms_is_toggle() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = pressed(t0, &ctx); + let e = m.on(Input::ShortcutUp { owner: A }, ms(t0, 299), &ctx); + assert!(!stops_capture(&e) && !transcribes(&e), "{e:?}"); + assert_eq!(recording_mode(&m).as_deref(), Some("toggle")); +} + +// ── R3.2: key repeat ignored, a fresh press after the release accepted ────── + +#[test] +fn state_repeated_shortcut_down_while_held_is_ignored() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = pressed(t0, &ctx); + // X11 auto-repeat arrives as fresh key-downs (is_held: false): none may toggle or stop. + for n in [30, 60, 90, 400, 900] { + let e = m.on(Input::ShortcutDown { owner: A }, ms(t0, n), &ctx); + assert!( + e.iter().all(|e| matches!(e, Effect::Render)), + "repeat at {n} ms: {e:?}" + ); + } + assert_eq!( + recording_mode(&m).as_deref(), + Some("pending"), + "still recording, not stopped" + ); + let e = m.on(Input::ShortcutUp { owner: A }, ms(t0, 1000), &ctx); + assert!( + stops_capture(&e) && transcribes(&e), + "the release after a long hold stops: {e:?}" + ); +} + +#[test] +fn state_fresh_press_after_release_stops_a_toggle() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = pressed(t0, &ctx); + m.on(Input::ShortcutUp { owner: A }, ms(t0, 100), &ctx); + let e = m.on(Input::ShortcutDown { owner: A }, ms(t0, 5000), &ctx); + assert!( + stops_capture(&e) && transcribes(&e), + "R2.3 a new press stops the toggle: {e:?}" + ); + assert!( + has(&e, |e| matches!( + e, + Effect::Transcribe { mode: Mode::Toggle } + )), + "tagged toggle: {e:?}" + ); +} + +#[test] +fn state_after_a_stop_the_next_press_starts_again() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = pressed(t0, &ctx); + m.on(Input::ShortcutUp { owner: A }, ms(t0, 800), &ctx); // hold stop + m.on( + Input::Transcribed { + text: "hello".into(), + }, + ms(t0, 900), + &ctx, + ); + assert!(is_idle(&m), "{:?}", m.phase()); + let e = m.on(Input::ShortcutDown { owner: A }, ms(t0, 1000), &ctx); + assert!( + starts_capture(&e), + "a fresh press after the release starts a new recording: {e:?}" + ); +} + +// R2.2 derived: a tap released before the device has opened must still toggle. +#[test] +fn state_tap_released_before_capture_started_still_toggles() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = Machine::default(); + m.on(Input::ShortcutDown { owner: A }, t0, &ctx); + m.on(Input::ShortcutUp { owner: A }, ms(t0, 80), &ctx); + m.on(Input::CaptureStarted, ms(t0, 120), &ctx); + assert!( + recording_mode(&m).is_some(), + "recording after the release: {:?}", + m.phase() + ); + let e = m.on(Input::ShortcutDown { owner: A }, ms(t0, 3000), &ctx); + assert!( + stops_capture(&e) && transcribes(&e), + "the next press stops it (toggle): {e:?}" + ); +} + +// ── R2.1 / R2.3: click ─────────────────────────────────────────────────────── + +#[test] +fn state_click_toggles_on_and_off() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + let e = m.on(Input::Click { owner: A }, ms(t0, 4000), &ctx); + assert!(stops_capture(&e) && transcribes(&e), "{e:?}"); +} + +#[test] +fn state_click_while_pending_stops() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = pressed(t0, &ctx); + let e = m.on(Input::Click { owner: A }, ms(t0, 100), &ctx); + assert!(stops_capture(&e) && transcribes(&e), "{e:?}"); +} + +#[test] +fn state_stray_shortcut_up_does_not_stop_a_click_recording() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + let e = m.on(Input::ShortcutUp { owner: A }, ms(t0, 500), &ctx); + assert!(!stops_capture(&e), "{e:?}"); + assert_eq!(recording_mode(&m).as_deref(), Some("toggle")); +} + +// ── R2.5: Escape cancels without transcribing ─────────────────────────────── + +#[test] +fn state_cancel_discards_without_transcription() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + let e = m.on(Input::Cancel { owner: A }, ms(t0, 1000), &ctx); + assert!(has(&e, |e| matches!(e, Effect::CancelCapture)), "{e:?}"); + assert!( + !transcribes(&e) && !stops_capture(&e), + "no transcription after a cancel: {e:?}" + ); + assert!(is_idle(&m)); +} + +#[test] +fn state_cancel_from_another_owner_keeps_the_recording() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + let e = m.on(Input::Cancel { owner: B }, ms(t0, 1000), &ctx); + assert!(!has(&e, |e| matches!(e, Effect::CancelCapture)), "{e:?}"); + assert_eq!(recording_mode(&m).as_deref(), Some("toggle")); +} + +// ── R3.3: focus loss ───────────────────────────────────────────────────────── + +#[test] +fn state_window_deactivated_during_hold_stops_and_transcribes() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = pressed(t0, &ctx); + let e = m.on(Input::WindowDeactivated { owner: A }, ms(t0, 700), &ctx); + assert!(stops_capture(&e) && transcribes(&e), "{e:?}"); +} + +#[test] +fn state_window_deactivated_during_toggle_keeps_recording() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + let e = m.on(Input::WindowDeactivated { owner: A }, ms(t0, 700), &ctx); + assert!(!stops_capture(&e) && !transcribes(&e), "{e:?}"); + assert_eq!(recording_mode(&m).as_deref(), Some("toggle")); +} + +// ── R2.6 / R9.1 / R9.2: capture events ────────────────────────────────────── + +#[test] +fn state_limit_stops_transcribes_and_says_so() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + let e = m.on(Input::CaptureLimit, ms(t0, 120_000), &ctx); + assert!(stops_capture(&e) && transcribes(&e), "{e:?}"); + let msg = notify_text(&e, A).expect("a notice for the owner"); + assert!(msg.contains("2-minute limit"), "{msg}"); +} + +#[test] +fn state_device_error_stops_transcribes_and_names_the_reason() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + let e = m.on( + Input::CaptureDeviceError("device unplugged".into()), + ms(t0, 3000), + &ctx, + ); + assert!(stops_capture(&e) && transcribes(&e), "{e:?}"); + let msg = notify_text(&e, A).expect("a notice"); + assert!( + msg.contains("The microphone stopped") && msg.contains("device unplugged"), + "{msg}" + ); +} + +#[test] +fn state_capture_failure_returns_to_idle_with_reason() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = Machine::default(); + m.on(Input::Click { owner: A }, t0, &ctx); + let e = m.on( + Input::CaptureFailed("no default input device".into()), + t0, + &ctx, + ); + let msg = notify_text(&e, A).expect("a notice"); + assert!( + msg.starts_with("No microphone available") && msg.contains("no default input device"), + "{msg}" + ); + assert!(!transcribes(&e)); + assert!(is_idle(&m)); +} + +// ── R4.6: the transcript ───────────────────────────────────────────────────── + +#[test] +fn state_transcribed_inserts_the_cleaned_text_for_the_owner() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + m.on(Input::Click { owner: A }, ms(t0, 2000), &ctx); + let e = m.on( + Input::Transcribed { + text: " Hello world. [BLANK_AUDIO] ".into(), + }, + ms(t0, 3000), + &ctx, + ); + let inserted = e.iter().find_map(|e| match e { + Effect::Insert { owner, text } if *owner == A => Some(text.to_string()), + _ => None, + }); + assert_eq!(inserted.as_deref(), Some("Hello world."), "{e:?}"); + assert!(is_idle(&m)); +} + +#[test] +fn state_empty_transcript_inserts_nothing_and_says_so() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + m.on(Input::Click { owner: A }, ms(t0, 2000), &ctx); + let e = m.on( + Input::Transcribed { + text: " [BLANK_AUDIO] (music) ".into(), + }, + ms(t0, 3000), + &ctx, + ); + assert!(!has(&e, |e| matches!(e, Effect::Insert { .. })), "{e:?}"); + assert_eq!( + notify_text(&e, A).as_deref(), + Some("No speech was recognised") + ); + assert!(is_idle(&m)); +} + +#[test] +fn state_transcription_failure_notifies_and_inserts_nothing() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + m.on(Input::Click { owner: A }, ms(t0, 2000), &ctx); + let e = m.on( + Input::TranscribeFailed("Remote transcription failed: HTTP 500".into()), + ms(t0, 3000), + &ctx, + ); + assert!(!has(&e, |e| matches!(e, Effect::Insert { .. }))); + assert!( + notify_text(&e, A).is_some_and(|m| m.contains("HTTP 500")), + "{e:?}" + ); + assert!(is_idle(&m)); +} + +// ── R5.1 / R5.2: one recording at a time ──────────────────────────────────── + +#[test] +fn state_other_owner_is_busy_while_recording() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + for input in [Input::Click { owner: B }, Input::ShortcutDown { owner: B }] { + let e = m.on(input, ms(t0, 500), &ctx); + assert!( + has(&e, |e| matches!(e, Effect::Busy { owner } if *owner == B)), + "{e:?}" + ); + assert!( + !starts_capture(&e) && !stops_capture(&e), + "A's recording untouched: {e:?}" + ); + } + assert!(matches!(m.phase(), Phase::Recording { owner, .. } if *owner == A)); +} + +#[test] +fn state_other_owner_is_busy_while_transcribing_and_owner_is_ignored() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = clicked(t0, &ctx); + m.on(Input::Click { owner: A }, ms(t0, 2000), &ctx); + let e = m.on(Input::Click { owner: B }, ms(t0, 2100), &ctx); + assert!( + has(&e, |e| matches!(e, Effect::Busy { owner } if *owner == B)), + "{e:?}" + ); + for input in [Input::Click { owner: A }, Input::ShortcutDown { owner: A }] { + let e = m.on(input, ms(t0, 2200), &ctx); + assert!( + e.iter().all(|e| matches!(e, Effect::Render)), + "R5.2 the owner's start is ignored while transcribing: {e:?}" + ); + } + assert!(matches!(m.phase(), Phase::Transcribing { owner } if *owner == A)); +} + +#[test] +fn state_other_owner_is_busy_while_awaiting_consent() { + let t0 = Instant::now(); + let ctx = Ctx { + model_present: false, + ..local_ready() + }; + let mut m = Machine::default(); + m.on(Input::Click { owner: A }, t0, &ctx); + let e = m.on(Input::Click { owner: B }, t0, &ctx); + assert!( + has(&e, |e| matches!(e, Effect::Busy { owner } if *owner == B)), + "{e:?}" + ); +} + +// ── R6.1 / R6.7 / R6.9 / R8.7: consent and download ───────────────────────── + +#[test] +fn state_missing_model_asks_consent_and_records_nothing() { + let t0 = Instant::now(); + let ctx = Ctx { + model_present: false, + ..local_ready() + }; + for input in [Input::Click { owner: A }, Input::ShortcutDown { owner: A }] { + let mut m = Machine::default(); + let e = m.on(input, t0, &ctx); + assert!(has(&e, |e| matches!(e, Effect::AskConsent)), "{e:?}"); + assert!( + !starts_capture(&e), + "no recording before the model exists: {e:?}" + ); + assert!( + !has(&e, |e| matches!(e, Effect::StartDownload)), + "R6.9 no download before consent" + ); + assert!(matches!(m.phase(), Phase::AwaitingConsent { owner } if *owner == A)); + } +} + +#[test] +fn state_remote_mode_never_asks_for_a_model() { + let t0 = Instant::now(); + let ctx = Ctx { + remote: true, + model_present: false, + ..local_ready() + }; + let mut m = Machine::default(); + let e = m.on(Input::Click { owner: A }, t0, &ctx); + assert!(starts_capture(&e), "{e:?}"); + assert!( + !has(&e, |e| matches!( + e, + Effect::AskConsent | Effect::StartDownload + )), + "{e:?}" + ); +} + +#[test] +fn state_consent_declined_downloads_nothing() { + let t0 = Instant::now(); + let ctx = Ctx { + model_present: false, + ..local_ready() + }; + let mut m = Machine::default(); + m.on(Input::Click { owner: A }, t0, &ctx); + let e = m.on(Input::ConsentDeclined, t0, &ctx); + assert!( + !has(&e, |e| matches!( + e, + Effect::StartDownload | Effect::StartCapture + )), + "{e:?}" + ); + assert!(is_idle(&m)); +} + +#[test] +fn state_consent_given_downloads_then_idles_without_recording() { + let t0 = Instant::now(); + let ctx = Ctx { + model_present: false, + ..local_ready() + }; + let mut m = Machine::default(); + m.on(Input::Click { owner: A }, t0, &ctx); + let e = m.on(Input::ConsentGiven, t0, &ctx); + assert!(has(&e, |e| matches!(e, Effect::StartDownload)), "{e:?}"); + m.on(Input::DownloadProgress(1_000), ms(t0, 300), &ctx); + assert!( + matches!(m.phase(), Phase::Downloading { owner, bytes } if *owner == A && *bytes == 1_000) + ); + let e = m.on(Input::DownloadDone, ms(t0, 9000), &ctx); + assert!( + !starts_capture(&e), + "recording is not started by the download: {e:?}" + ); + assert!(is_idle(&m)); +} + +#[test] +fn state_download_cancel_and_failure_return_to_idle() { + let t0 = Instant::now(); + let ctx = Ctx { + model_present: false, + ..local_ready() + }; + let mut m = Machine::default(); + m.on(Input::Click { owner: A }, t0, &ctx); + m.on(Input::ConsentGiven, t0, &ctx); + let e = m.on(Input::DownloadCancel, ms(t0, 100), &ctx); + assert!(has(&e, |e| matches!(e, Effect::CancelDownload)), "{e:?}"); + assert!(is_idle(&m), "R6.6 idle after a cancel: {:?}", m.phase()); + + let mut m = Machine::default(); + m.on(Input::Click { owner: A }, t0, &ctx); + m.on(Input::ConsentGiven, t0, &ctx); + let e = m.on(Input::DownloadFailed("HTTP 404".into()), ms(t0, 100), &ctx); + assert!( + notify_text(&e, A).is_some_and(|m| m.contains("HTTP 404")), + "{e:?}" + ); + assert!(is_idle(&m)); +} + +// ── R1.2 / R1.3: disabled and read-only start nothing ─────────────────────── + +#[test] +fn state_disabled_or_read_only_starts_nothing() { + let t0 = Instant::now(); + for ctx in [ + Ctx { + enabled: false, + ..local_ready() + }, + Ctx { + composer_read_only: true, + ..local_ready() + }, + ] { + let mut m = Machine::default(); + for input in [Input::Click { owner: A }, Input::ShortcutDown { owner: A }] { + let e = m.on(input, t0, &ctx); + assert!( + !starts_capture(&e) && !has(&e, |e| matches!(e, Effect::AskConsent)), + "{e:?}" + ); + } + assert!(is_idle(&m)); + } +} + +// Inputs that do not apply are no-ops, never panics. +#[test] +fn state_inputs_out_of_phase_are_ignored() { + let t0 = Instant::now(); + let ctx = local_ready(); + let mut m = Machine::default(); + for input in [ + Input::ShortcutUp { owner: A }, + Input::Cancel { owner: A }, + Input::WindowDeactivated { owner: A }, + Input::CaptureStarted, + Input::CaptureLimit, + Input::CaptureDeviceError("x".into()), + Input::Transcribed { + text: "stray".into(), + }, + Input::TranscribeFailed("x".into()), + Input::ConsentGiven, + Input::ConsentDeclined, + Input::DownloadProgress(5), + Input::DownloadDone, + Input::DownloadFailed("x".into()), + Input::DownloadCancel, + ] { + let e = m.on(input, t0, &ctx); + assert!( + !has(&e, |e| matches!( + e, + Effect::Insert { .. } | Effect::StartCapture | Effect::StartDownload + )), + "{e:?}" + ); + assert!(is_idle(&m)); + } +} diff --git a/crates/voice_input/src/transcript.rs b/crates/voice_input/src/transcript.rs new file mode 100644 index 0000000..c4394f2 --- /dev/null +++ b/crates/voice_input/src/transcript.rs @@ -0,0 +1,56 @@ +//! Transcript cleaning and spacing before insertion into the composer (R4.2, R4.3). + +/// Removes whisper's non-speech annotations (`[BLANK_AUDIO]`, `[Music]`, `(music)`, +/// `(silence)`), collapses runs of whitespace to one space and trims (R4.3). +/// +/// A bracketed or parenthesised span counts as an annotation only when it stands alone in a +/// segment: it opens at the start of the transcript, after sentence-ending punctuation or after +/// another annotation, and it is followed by whitespace or the end. Brackets that are part of +/// speech ("array [i]", "f(x)") are kept. +pub fn clean(raw: &str) -> String { + let mut kept = String::with_capacity(raw.len()); + let mut at_segment_start = true; + let mut rest = raw; + while let Some(c) = rest.chars().next() { + if at_segment_start && let Some(len) = annotation_len(rest) { + rest = &rest[len..]; + continue; + } + if !c.is_whitespace() { + at_segment_start = matches!(c, '.' | '!' | '?' | '…'); + } + kept.push(c); + rest = &rest[c.len_utf8()..]; + } + kept.split_whitespace().collect::>().join(" ") +} + +/// Byte length of the annotation `text` starts with: a `[...]` or `(...)` span with non-blank, +/// unnested content, followed by whitespace or the end of `text`. +fn annotation_len(text: &str) -> Option { + let close = match text.chars().next()? { + '[' => ']', + '(' => ')', + _ => return None, + }; + let inner_end = 1 + text[1..].find(close)?; + let inner = &text[1..inner_end]; + if inner.trim().is_empty() || inner.contains(['[', '(', ']', ')']) { + return None; + } + let len = inner_end + close.len_utf8(); + match text[len..].chars().next() { + None => Some(len), + Some(next) if next.is_whitespace() => Some(len), + Some(_) => None, + } +} + +/// Prefixes one space when the character before the cursor is not whitespace (R4.2). An empty +/// transcript stays empty, so nothing is inserted (R4.6). +pub fn for_insertion(previous: Option, cleaned: &str) -> String { + match previous { + Some(c) if !c.is_whitespace() && !cleaned.is_empty() => format!(" {cleaned}"), + _ => cleaned.to_string(), + } +} diff --git a/crates/voice_input/src/transcript_tests.rs b/crates/voice_input/src/transcript_tests.rs new file mode 100644 index 0000000..377f823 --- /dev/null +++ b/crates/voice_input/src/transcript_tests.rs @@ -0,0 +1,97 @@ +// Story 044, task 2.2 — transcript cleaning and spacing (R4.2, R4.3, R4.6). Wired in the crate root +// `src/voice_input.rs` as +// +// #[cfg(test)] +// mod transcript_tests; +// +// Validation filter: `transcript` (every test path is `transcript_tests::…`). +// +// Contract (design.md C1.6): `transcript::clean(&str) -> String`, +// `transcript::for_insertion(Option, &str) -> String`. +// +// Hostile halves: an annotation standing alone is removed AND brackets that are part of speech +// ("array [i]", "f(x)") survive; one space after a letter or punctuation AND none after whitespace, +// a newline or at the start of the composer; Unicode letters count as non-whitespace and a +// non-breaking space counts as whitespace. + +use crate::transcript::{clean, for_insertion}; + +#[test] +fn transcript_trims_and_collapses_whitespace() { + assert_eq!(clean(" Hello world.\n\t"), "Hello world."); + assert_eq!(clean(""), ""); + assert_eq!(clean(" \n "), ""); +} + +#[test] +fn transcript_removes_standalone_annotations() { + assert_eq!(clean("[BLANK_AUDIO]"), ""); + assert_eq!(clean("(music)"), ""); + assert_eq!(clean(" [Music] Hello there. (silence) "), "Hello there."); + assert_eq!(clean("Hello. [BLANK_AUDIO] World."), "Hello. World."); + assert_eq!( + clean("[BLANK_AUDIO] (music) [Music]"), + "", + "R4.6 nothing but annotations is empty" + ); +} + +#[test] +fn transcript_keeps_brackets_that_are_speech() { + assert_eq!( + clean("index the array [i] please"), + "index the array [i] please" + ); + assert_eq!(clean("call f(x) twice"), "call f(x) twice"); +} + +#[test] +fn transcript_keeps_unicode_text() { + assert_eq!( + clean(" Olá, ação concluída. [BLANK_AUDIO] "), + "Olá, ação concluída." + ); + assert_eq!(clean("日本語 テスト"), "日本語 テスト"); +} + +#[test] +fn transcript_spacing_after_non_whitespace() { + assert_eq!( + for_insertion(Some('a'), "hello"), + " hello", + "R4.2 after a letter" + ); + assert_eq!( + for_insertion(Some('.'), "hello"), + " hello", + "after punctuation" + ); + assert_eq!( + for_insertion(Some('ç'), "hello"), + " hello", + "after a non-ASCII letter" + ); +} + +#[test] +fn transcript_no_spacing_after_whitespace_or_at_start() { + assert_eq!( + for_insertion(None, "hello"), + "hello", + "start of the composer" + ); + assert_eq!(for_insertion(Some(' '), "hello"), "hello"); + assert_eq!(for_insertion(Some('\n'), "hello"), "hello"); + assert_eq!(for_insertion(Some('\t'), "hello"), "hello"); + assert_eq!( + for_insertion(Some('\u{a0}'), "hello"), + "hello", + "NBSP is whitespace" + ); +} + +#[test] +fn transcript_spacing_adds_exactly_one_space() { + assert_eq!(for_insertion(Some('x'), "a b"), " a b"); + assert!(!for_insertion(Some('x'), "a").starts_with(" ")); +} diff --git a/crates/voice_input/src/voice_input.rs b/crates/voice_input/src/voice_input.rs new file mode 100644 index 0000000..8ad6e1f --- /dev/null +++ b/crates/voice_input/src/voice_input.rs @@ -0,0 +1,1321 @@ +//! Dictation into the agent composer. + +pub mod audio_prep; +pub mod capture; +pub mod download; +pub mod local; +pub mod models; +pub mod remote; +mod settings; +pub mod state; +pub mod transcript; + +pub use crate::models::VoiceModel; +pub use crate::settings::{Language, RemoteSettings, VoiceInputSettings}; + +use std::{ + collections::HashMap, + fs, io, + path::PathBuf, + sync::{Arc, Mutex, PoisonError}, + time::{Duration, Instant}, +}; + +use ::settings::{Settings as _, SettingsStore}; +use audio::AudioSettings; +use futures::channel::oneshot; +use gpui::{ + AnyWeakEntity, App, AppContext as _, AsyncApp, Context, Entity, EntityId, EventEmitter, Global, + SharedString, Subscription, Task, actions, +}; +use http_client::HttpClient; + +use crate::capture::{ActiveCapture, CaptureBackend, CaptureEvent, Recording}; +use crate::download::{DownloadError, download_model, model_name}; +use crate::local::{LocalError, LocalTranscriber}; +use crate::models::{CachedLookup, HashCache, ModelOrigin, ModelSpec, ResolvedModel}; +use crate::state::{Ctx, Effect, Input, Machine, Mode, Owner, Phase, TAP_WINDOW}; + +actions!( + voice_input, + [ + /// Starts recording, or stops it and transcribes what was said. + ToggleRecording, + /// Stops recording and discards the audio. + CancelRecording, + /// Declines the speech model download the voice strip asks for. + DismissDownload + ] +); + +/// How often the recording's elapsed time is re-rendered. +const TICK: Duration = Duration::from_secs(1); +/// How often an idle local model is offered the chance to leave memory. +const IDLE_CHECK: Duration = Duration::from_secs(60); +/// Room for both early-end events of one recording (`Limit` and `DeviceError`). +const CAPTURE_EVENT_CAPACITY: usize = 4; + +const BUSY: &str = "Voice input is busy in another thread"; +const NO_SPEECH: &str = "No speech was recognised"; + +/// Registers the `voice_input` settings section and creates the app-wide [`VoiceInput`], with +/// models downloaded to the user's data directory and the real microphone. +pub fn init(http: Arc, cx: &mut App) { + let dirs = ModelDirs { + user: models::models_dir(), + system: models::system_models_dir(), + }; + init_with(http, dirs, Arc::new(capture::Cpal), cx); +} + +/// [`init`] with the models directory and the capture backend passed in, so a test never writes +/// into the real data directory nor opens the operator's microphone. Packaged models are looked +/// up in `/system`, never in the system's own directory, so an installed +/// `whisper-models` package cannot change what a test sees. +pub fn init_in( + http: Arc, + models_dir: PathBuf, + capture: Arc, + cx: &mut App, +) { + let dirs = ModelDirs { + system: models_dir.join("system"), + user: models_dir, + }; + init_with(http, dirs, capture, cx); +} + +struct ModelDirs { + /// Read-only models installed by the package manager, preferred (R6.10). + system: PathBuf, + /// Where models are downloaded. + user: PathBuf, +} + +fn init_with( + http: Arc, + dirs: ModelDirs, + capture: Arc, + cx: &mut App, +) { + VoiceInputSettings::register(cx); + local::install_logging(); + let entity = cx.new(|cx| VoiceInput::new(http, dirs, capture, cx)); + cx.set_global(GlobalVoiceInput(entity)); +} + +struct GlobalVoiceInput(Entity); + +impl Global for GlobalVoiceInput {} + +/// What a composer's microphone button and strip render. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum ViewState { + /// Voice input is disabled in the settings (R1.2). + Hidden, + /// Ready; names the remote host when audio would leave the machine (R8.3). + Idle { + remote_host: Option, + }, + /// Recording, transcribing or downloading for another composer (R5.1). + Busy, + /// The model is missing and the user is asked before anything is downloaded (R6.1). + /// `size` is the download size as shown, e.g. `148 MB`. + AwaitingConsent { + model: VoiceModel, + size: SharedString, + }, + Downloading { + percent: u8, + }, + Recording { + elapsed: Duration, + }, + /// Transcribing, or still opening the device: the button is disabled either way. + Transcribing, +} + +/// What the entity tells the composers. `owner` is the thread view the request came from. +#[derive(Clone, Debug)] +pub enum VoiceEvent { + Insert { + owner: EntityId, + text: String, + }, + Notify { + owner: EntityId, + message: String, + }, + /// The transcript was dropped because its composer had been closed (R4.7). + Discarded { + owner: EntityId, + }, + /// Something every view shows has changed: re-render. + Changed, +} + +#[derive(Default)] +struct OwnerEntry { + /// `None` until the thread view registers itself; an unregistered owner counts as alive. + alive: Option, + read_only: bool, +} + +/// The R10.1 line of the recording in flight, filled as the recording moves along. +struct RecordingLog { + mode: &'static str, + backend: &'static str, + model: String, + host: Option, + audio_ms: u128, + transcribe_ms: u128, +} + +impl RecordingLog { + fn emit(&self, outcome: &str) { + let host = self + .host + .as_ref() + .map(|host| format!(" host={host}")) + .unwrap_or_default(); + let level = if outcome == "error" { + log::Level::Warn + } else { + log::Level::Info + }; + log::log!( + level, + "voice_input event=recording mode={} audio_ms={} backend={} model={} transcribe_ms={} outcome={outcome}{host}", + self.mode, + self.audio_ms, + self.backend, + self.model, + self.transcribe_ms, + ); + } +} + +/// The app-wide voice input: runs the [`Machine`]'s effects and feeds every async result back +/// into it as an [`Input`]. +pub struct VoiceInput { + machine: Machine, + settings: VoiceInputSettings, + /// The settings the current recording started under: it is transcribed by the backend it + /// was recorded for, whatever changed since (R7.1, R8.7). + recording_settings: Option, + /// The remote host of the recording in flight, kept until the next start so the button names + /// it while transcribing too (R8.3), after `recording_settings` has been handed over. + in_flight_host: Option, + http: Arc, + /// A client that never uses a proxy, for plain-http (hence loopback) remote URLs (R8.9). + loopback_http: Option>, + models_dir: PathBuf, + system_models_dir: PathBuf, + hash_cache: Arc, + /// Hashes a packaged model on the background executor; inputs wait in `deferred` meanwhile. + hash_check: Option>, + /// Inputs held, in order, until `hash_check` answers whether the packaged model is valid. + deferred: Vec<(Input, Instant)>, + /// Set while the held inputs are replayed: they no longer wait for a hash. + replaying: bool, + /// Set while the machine is told to drop the work of a composer that was closed. + abandoning: bool, + capture: Arc, + /// Bumped on every capture start, so a late result or event of an older start is ignored. + capture_generation: u64, + active: Option>, + /// Resolves once the last stopped or cancelled capture has released its device; the next + /// start waits for it. + capture_closed: Option>, + capture_opening: Option>, + capture_events: Option>, + /// The recording being handed back by `ActiveCapture::stop`, for the transcription. + stopping: Option>, + transcriber: Arc>, + transcription: Option>, + download_cancel: Option>, + download: Option<(Task<()>, Task<()>)>, + /// The model being downloaded, which the progress is measured against. + download_spec: Option<&'static ModelSpec>, + owners: HashMap, + recording_log: Option, + /// The R10.1 line of a transcript just handed to its composer, held until the next effect + /// flush so the composer can report a refusal (R10.1 `outcome=refused`). + pending_insert_log: Option<(Owner, RecordingLog)>, + ticker: Option>, + _idle_release: Task<()>, + _settings: Subscription, +} + +impl EventEmitter for VoiceInput {} + +impl VoiceInput { + /// The entity [`init`] created. + /// + /// Panics when [`init`] has not run; use [`VoiceInput::try_global`] where that can happen. + pub fn global(cx: &App) -> Entity { + cx.global::().0.clone() + } + + pub fn try_global(cx: &App) -> Option> { + cx.try_global::() + .map(|global| global.0.clone()) + } + + /// Sets the proxy-less client a plain-http (hence loopback) remote URL is sent through, so it + /// goes direct even when the app's client has a proxy (R8.9). Without one, plain http behind + /// a proxy is refused. + pub fn set_loopback_http(&mut self, http: Arc) { + self.loopback_http = Some(http); + } + + fn new( + http: Arc, + dirs: ModelDirs, + capture: Arc, + cx: &mut Context, + ) -> Self { + let transcriber = Arc::new(Mutex::new(LocalTranscriber::default())); + let idle_transcriber = transcriber.clone(); + let idle_release = cx.spawn(async move |_, cx| { + loop { + cx.background_executor().timer(IDLE_CHECK).await; + // A transcription in progress holds the lock and is, by definition, not idle. + if let Ok(mut transcriber) = idle_transcriber.try_lock() { + transcriber.release_if_idle(Instant::now()); + } + } + }); + let settings_subscription = cx.observe_global::(|this, cx| { + let settings = VoiceInputSettings::get_global(cx).clone(); + if settings != this.settings { + this.settings_changed(settings, cx); + } + }); + // The user's directory only: the system one is the package manager's (R6.12). + let staging_dir = dirs.user.clone(); + cx.background_spawn(async move { + download::sweep_staging(&staging_dir, 2 * download::STALL_TIMEOUT); + }) + .detach(); + let mut this = Self { + machine: Machine::default(), + settings: VoiceInputSettings::get_global(cx).clone(), + recording_settings: None, + in_flight_host: None, + http, + loopback_http: None, + models_dir: dirs.user, + system_models_dir: dirs.system, + hash_cache: Arc::default(), + hash_check: None, + deferred: Vec::new(), + replaying: false, + abandoning: false, + capture, + capture_generation: 0, + active: None, + capture_closed: None, + capture_opening: None, + capture_events: None, + stopping: None, + transcriber, + transcription: None, + download_cancel: None, + download: None, + download_spec: None, + owners: HashMap::default(), + recording_log: None, + pending_insert_log: None, + ticker: None, + _idle_release: idle_release, + _settings: settings_subscription, + }; + this.warm_hash_cache(cx); + this + } + + fn settings_changed(&mut self, settings: VoiceInputSettings, cx: &mut Context) { + let previous = std::mem::replace(&mut self.settings, settings); + if previous.enabled && !self.settings.enabled { + // The button is hidden and the shortcut unbound: nothing could stop this recording + // or answer this prompt any more (R1.2). + self.abandon_current_work(cx); + } + let needs_local_model = self.settings.enabled && !self.settings.remote.enabled; + let needed_before = previous.enabled && !previous.remote.enabled; + if needs_local_model && (!needed_before || previous.model != self.settings.model) { + self.warm_hash_cache(cx); + } + cx.emit(VoiceEvent::Changed); + } + + /// The SHA-256 verdict cache for packaged models, for tests that count hashes. + #[cfg(test)] + pub(crate) fn hash_cache(&self) -> &HashCache { + &self.hash_cache + } + + /// Records the thread view behind `owner`, so a transcript finishing after it closed is + /// discarded rather than delivered to nobody (R4.7). + pub fn register_owner(&mut self, owner: Owner, alive: AnyWeakEntity) { + // Closed composers are forgotten only while nothing is in flight: an in-flight owner's + // entry is what tells its transcript to be discarded. + if matches!(self.machine.phase(), Phase::Idle) { + self.owners.retain(|_, entry| { + entry + .alive + .as_ref() + .is_none_or(|alive| alive.upgrade().is_some()) + }); + } + self.owners.entry(owner).or_default().alive = Some(alive); + } + + /// Records whether `owner`'s composer is read-only; a read-only composer starts nothing + /// (R1.3). + pub fn set_composer_read_only(&mut self, owner: Owner, read_only: bool) { + self.owners.entry(owner).or_default().read_only = read_only; + } + + /// Feeds one input to the machine and runs the effects it returns. + pub fn handle(&mut self, input: Input, cx: &mut Context) { + self.handle_at(input, Instant::now(), cx); + } + + /// `now` is when the input arrived: a held input replays with it, so a press keeps the + /// time it was made (C1.9) however long the hash took. + fn handle_at(&mut self, input: Input, now: Instant, cx: &mut Context) { + if !self.deferred.is_empty() { + // Keeps the order: a key-up must not overtake the key-down held before it. + self.deferred.push((input, now)); + return; + } + if let Input::Click { owner } | Input::ShortcutDown { owner } = input { + // A composer closed while it held the machine would otherwise make every other + // composer busy until a restart. + if self.abandon_closed_owner(Some(owner), cx) { + return; + } + if self.needs_hash_check() { + self.deferred.push((input, now)); + self.warm_hash_cache(cx); + return; + } + } + self.run_at(input, now, cx); + } + + fn run(&mut self, input: Input, cx: &mut Context) { + self.run_at(input, Instant::now(), cx); + } + + fn run_at(&mut self, input: Input, now: Instant, cx: &mut Context) { + let finishing = matches!( + input, + Input::Transcribed { .. } | Input::TranscribeFailed(_) + ) && matches!(self.machine.phase(), Phase::Transcribing { .. }); + // A download kept after its composer closed, and never adopted, ends with nobody to + // tell (R6.14); download.rs has already logged how it ended (R10.2). + let unowned_download_failed = matches!(input, Input::DownloadFailed(_)) + && matches!( + self.machine.phase(), + Phase::Downloading { owner, .. } if self.owner_closed(*owner) + ); + let before = self.machine.phase().clone(); + let ctx = self.ctx(&input); + let effects = self.machine.on(input, now, &ctx); + + let mut outcome = None; + for effect in effects { + match effect { + Effect::StartCapture => self.start_capture(cx), + Effect::StopCapture => self.stop_capture(cx), + Effect::CancelCapture => { + self.cancel_capture(cx); + let outcome = if self.abandoning { + "discarded" + } else { + "cancelled" + }; + self.log_cancelled(&before, now, outcome); + } + Effect::AskConsent | Effect::Render => cx.emit(VoiceEvent::Changed), + Effect::StartDownload => self.start_download(cx), + Effect::CancelDownload => { + if let Some(cancel) = self.download_cancel.take() + && cancel.send(()).is_err() + { + log::debug!("voice_input event=download_cancel_unheard"); + } + } + Effect::Transcribe { mode } => self.transcribe(mode, cx), + Effect::Insert { owner, text } => { + // A thread view held by a parent whose last handle was dropped in this same + // update still upgrades until the flush releases the parent. The decision is + // deferred past that flush so such a composer is seen as closed (R4.7). The + // handle is taken now: the machine is already `Idle`, so a `register_owner` + // in the gap may prune the owner's entry. + let alive = self + .owners + .get(&owner) + .and_then(|entry| entry.alive.clone()); + let recording_log = if finishing { + self.recording_log.take() + } else { + None + }; + let this = cx.weak_entity(); + cx.defer(move |cx| { + let delivered = this.update(cx, |this, cx| { + this.deliver_transcript(owner, alive, text, recording_log, cx) + }); + if delivered.is_err() { + log::debug!( + "voice_input event=transcript_dropped reason=voice_input_released" + ); + } + }); + } + Effect::Notify { owner, .. } if unowned_download_failed => { + self.owners.remove(&owner); + log::info!("voice_input event=download_notice_dropped reason=owner_closed"); + } + Effect::Notify { owner, message } => { + if finishing { + outcome = Some(if message == NO_SPEECH { + "empty" + } else { + "error" + }); + } + cx.emit(VoiceEvent::Notify { + owner: EntityId::from(owner.0), + message, + }); + } + Effect::Busy { owner } => cx.emit(VoiceEvent::Notify { + owner: EntityId::from(owner.0), + message: BUSY.to_string(), + }), + } + } + + if finishing && let Some(log) = self.recording_log.take() { + log.emit(outcome.unwrap_or("error")); + } + self.sync_ticker(cx); + } + + /// Hands a finished transcript to its composer, or discards it when the composer is gone. + /// + /// `alive` is the owner's handle as registered when the transcript arrived; `None` (never + /// registered) counts as open. + fn deliver_transcript( + &mut self, + owner: Owner, + alive: Option, + text: String, + recording_log: Option, + cx: &mut Context, + ) { + let id = EntityId::from(owner.0); + let closed = alive.is_some_and(|alive| alive.upgrade().is_none()); + if closed { + self.owners.remove(&owner); + cx.emit(VoiceEvent::Discarded { owner: id }); + if let Some(recording_log) = recording_log { + recording_log.emit("discarded"); + } + return; + } + cx.emit(VoiceEvent::Insert { owner: id, text }); + let Some(recording_log) = recording_log else { + return; + }; + // The composer answers in its `Insert` handler, which runs in the flush this emit is + // part of; the fallback is queued behind it, so it sees whether a refusal was reported. + self.flush_pending_insert_log(); + self.pending_insert_log = Some((owner, recording_log)); + let this = cx.weak_entity(); + cx.defer(move |cx| { + let flushed = this.update(cx, |this, _| this.flush_pending_insert_log()); + if flushed.is_err() { + log::debug!("voice_input event=insert_log_dropped reason=voice_input_released"); + } + }); + } + + /// Logs a pending R10.1 line nobody refused as `outcome=inserted`. + fn flush_pending_insert_log(&mut self) { + if let Some((_, recording_log)) = self.pending_insert_log.take() { + recording_log.emit("inserted"); + } + } + + /// Called by the composer whose `Insert` handler could not take the transcript (it turned + /// read-only while the transcript was on its way): the recording's R10.1 line says + /// `outcome=refused` instead of `inserted`. Does nothing for another owner, or when no line + /// is pending. The line carries no transcript or error text (R10.3). + pub fn report_insert_refused(&mut self, owner: Owner) { + match self.pending_insert_log.take() { + Some((pending, recording_log)) if pending == owner => recording_log.emit("refused"), + other => self.pending_insert_log = other, + } + } + + /// Whether the `voice_input.enabled` setting is on: the composer binds the voice shortcut + /// only then (R1.2). + pub fn is_enabled(&self) -> bool { + self.settings.enabled + } + + /// What `owner`'s button and strip show. + pub fn view_state(&self, owner: EntityId) -> ViewState { + if !self.settings.enabled { + return ViewState::Hidden; + } + let owner = Owner(owner.as_u64()); + match self.machine.phase() { + Phase::Idle => ViewState::Idle { + remote_host: settings_remote_host(&self.settings), + }, + // A download kept after its composer closed belongs to whoever looks at it: every + // composer shows its progress and Cancel (R6.14, R6.15). A busy button would take no + // click, so the mouse could never reach it. + Phase::Downloading { owner: current, .. } + if *current != owner && !self.owner_closed(*current) => + { + ViewState::Busy + } + Phase::AwaitingConsent { owner: current } + | Phase::Starting { owner: current, .. } + | Phase::Recording { owner: current, .. } + | Phase::Transcribing { owner: current } + if *current != owner => + { + ViewState::Busy + } + Phase::AwaitingConsent { .. } => { + let spec = models::spec(self.settings.model); + ViewState::AwaitingConsent { + model: self.settings.model, + size: models::human_size(spec.size).into(), + } + } + Phase::Downloading { bytes, .. } => { + let spec = self + .download_spec + .unwrap_or_else(|| models::spec(self.settings.model)); + let size = spec.size.max(1); + let percent = bytes.saturating_mul(100) / size; + ViewState::Downloading { + percent: u8::try_from(percent.min(100)).unwrap_or(100), + } + } + Phase::Recording { started, .. } => ViewState::Recording { + elapsed: Instant::now().saturating_duration_since(*started), + }, + Phase::Starting { .. } | Phase::Transcribing { .. } => ViewState::Transcribing, + } + } + + /// The host audio goes to, or `None` in local mode (R8.3). While a recording is in flight it + /// is that recording's host, since it is transcribed by the backend it was recorded for + /// (R8.7) whatever the settings say now. + pub fn remote_host(&self) -> Option { + match self.machine.phase() { + Phase::Starting { .. } | Phase::Recording { .. } | Phase::Transcribing { .. } => { + self.in_flight_host.clone() + } + Phase::Idle | Phase::AwaitingConsent { .. } | Phase::Downloading { .. } => { + settings_remote_host(&self.settings) + } + } + } + + fn ctx(&self, input: &Input) -> Ctx { + let owner = match input { + Input::Click { owner } | Input::ShortcutDown { owner } => Some(*owner), + _ => None, + }; + let remote = self.settings.remote.enabled; + // Only a start input reads the model's presence, and remote mode never needs one (R8.7). + let model_present = owner.is_some() && !remote && self.model_present_cached(); + let composer_read_only = owner + .and_then(|owner| self.owners.get(&owner)) + .is_some_and(|entry| entry.read_only); + Ctx { + enabled: self.settings.enabled, + remote, + model_present, + composer_read_only, + } + } + + fn lookup_model_cached(&self) -> CachedLookup { + models::lookup_model_cached( + models::spec(self.settings.model), + &self.system_models_dir, + &self.models_dir, + &self.hash_cache, + ) + } + + /// Never hashes: it runs inside `handle`, on the UI thread. An unverified packaged file only + /// reaches here while held inputs are replayed after a hash that did not settle it; it + /// counts by its size, and the transcription re-checks it off the UI thread. + fn model_present_cached(&self) -> bool { + match self.lookup_model_cached() { + CachedLookup::Resolved(resolved) => resolved.is_some(), + CachedLookup::Unverified => true, + } + } + + /// Whether a start input must wait for a packaged model to be hashed: the machine is idle, a + /// local model is needed, and the packaged file changed since it was last hashed (or never + /// was) — on first use, after a package upgrade, after a switch from remote to local. + fn needs_hash_check(&self) -> bool { + !self.replaying + && self.settings.enabled + && !self.settings.remote.enabled + && matches!(self.machine.phase(), Phase::Idle) + && self.lookup_model_cached() == CachedLookup::Unverified + } + + /// Hashes the configured packaged model on the background executor, at most one at a time, + /// then replays the inputs that waited for it. + fn warm_hash_cache(&mut self, cx: &mut Context) { + if self.hash_check.is_some() || self.settings.remote.enabled { + return; + } + let spec = models::spec(self.settings.model); + let system_dir = self.system_models_dir.clone(); + let models_dir = self.models_dir.clone(); + let cache = self.hash_cache.clone(); + let hashing = cx.background_spawn(async move { + models::resolve_model_in(spec, &system_dir, &models_dir, &cache); + }); + self.hash_check = Some(cx.spawn(async move |this, cx| { + hashing.await; + this.update(cx, |this, cx| { + this.hash_check = None; + this.replay_deferred(cx); + }) + .ok(); + })); + } + + fn replay_deferred(&mut self, cx: &mut Context) { + let deferred = std::mem::take(&mut self.deferred); + self.replaying = true; + for (input, arrived) in deferred { + self.handle_at(input, arrived, cx); + } + self.replaying = false; + } + + /// Drops the consent prompt or recording of a composer that has been closed. A running + /// download is kept instead (R6.14): `starter`, the composer whose start input is being + /// handled, adopts it (R6.15). Returns whether that start input was consumed by the adoption. + fn abandon_closed_owner(&mut self, starter: Option, cx: &mut Context) -> bool { + let owner = match self.machine.phase() { + Phase::AwaitingConsent { owner } + | Phase::Downloading { owner, .. } + | Phase::Recording { owner, .. } => *owner, + _ => return false, + }; + let downloading = matches!(self.machine.phase(), Phase::Downloading { .. }); + // Without a starter the download simply runs on, and the closed owner's entry stays: it + // is what keeps the download's end from notifying anybody. + if !self.owner_closed(owner) || (downloading && starter.is_none()) { + return false; + } + log::info!( + "voice_input event=owner_closed phase={}", + phase_name(self.machine.phase()) + ); + let adopted = match starter { + Some(starter) if downloading => { + log::info!("voice_input event=download_adopted"); + self.run(Input::AdoptDownload { owner: starter }, cx); + true + } + _ => { + self.abandon_current_work(cx); + false + } + }; + self.owners.remove(&owner); + adopted + } + + /// Ends the consent prompt, download or recording in progress with nothing inserted, through + /// the inputs the machine already has for it. + fn abandon_current_work(&mut self, cx: &mut Context) { + let input = match self.machine.phase() { + Phase::AwaitingConsent { .. } => Input::ConsentDeclined, + Phase::Downloading { .. } => Input::DownloadCancel, + Phase::Recording { owner, .. } => Input::Cancel { owner: *owner }, + // A device still opening is cancelled once it has opened (`capture_opened`); a + // transcription finishes, and its insert is discarded if its composer is gone (R4.7). + Phase::Idle | Phase::Starting { .. } | Phase::Transcribing { .. } => return, + }; + self.abandoning = true; + self.run(input, cx); + self.abandoning = false; + } + + fn recording_owner_closed(&self) -> bool { + match self.machine.phase() { + Phase::Recording { owner, .. } => self.owner_closed(*owner), + _ => false, + } + } + + fn owner_closed(&self, owner: Owner) -> bool { + self.owners + .get(&owner) + .and_then(|entry| entry.alive.as_ref()) + .is_some_and(|alive| alive.upgrade().is_none()) + } + + fn start_capture(&mut self, cx: &mut Context) { + self.recording_settings = Some(self.settings.clone()); + self.in_flight_host = settings_remote_host(&self.settings); + let device = AudioSettings::get_global(cx).input_audio_device.clone(); + self.capture_generation += 1; + let generation = self.capture_generation; + let backend = self.capture.clone(); + let previous = self.capture_closed.take(); + let (events_tx, events_rx) = smol::channel::bounded(CAPTURE_EVENT_CAPACITY); + // `start` blocks until the device is open (or for up to its open timeout). + let opening = cx.background_spawn(async move { + if let Some(previous) = previous { + // Either answer means the previous capture has released the device. + previous.await.ok(); + } + backend.start(device, events_tx) + }); + self.capture_opening = Some(cx.spawn(async move |this, cx| { + let result = opening.await; + this.update(cx, |this, cx| { + this.capture_opened(generation, result, events_rx, cx) + }) + .ok(); + })); + } + + fn capture_opened( + &mut self, + generation: u64, + result: anyhow::Result>, + events: smol::channel::Receiver, + cx: &mut Context, + ) { + let starting = matches!(self.machine.phase(), Phase::Starting { .. }); + if generation != self.capture_generation || !starting { + if let Ok(active) = result { + self.release_capture(active, cx); + } + return; + } + match result { + Ok(active) => { + self.active = Some(active); + self.capture_events = Some(cx.spawn(async move |this, cx| { + while let Ok(event) = events.recv().await { + let input = match event { + CaptureEvent::Limit => Input::CaptureLimit, + CaptureEvent::DeviceError(reason) => Input::CaptureDeviceError(reason), + }; + let delivered = this.update(cx, |this, cx| { + if this.capture_generation == generation { + this.handle(input, cx); + } + }); + if delivered.is_err() { + break; + } + } + })); + self.run(Input::CaptureStarted, cx); + if !self.settings.enabled { + // Disabled while the device was opening. + self.abandon_current_work(cx); + } + } + Err(error) => self.run(Input::CaptureFailed(format!("{error:#}")), cx), + } + } + + /// `stop` joins the capture thread, so it runs off the UI thread. + fn stop_capture(&mut self, cx: &mut Context) { + let Some(active) = self.active.take() else { + return; + }; + let (closed_tx, closed_rx) = oneshot::channel(); + self.capture_closed = Some(closed_rx); + self.stopping = Some(cx.background_spawn(async move { + let recording = active.stop(); + closed_tx.send(()).ok(); + recording + })); + } + + fn cancel_capture(&mut self, cx: &mut Context) { + if let Some(active) = self.active.take() { + self.release_capture(active, cx); + } + } + + /// Cancels `active` off the UI thread; the next start waits until it has let go. + fn release_capture(&mut self, active: Box, cx: &mut Context) { + let (closed_tx, closed_rx) = oneshot::channel(); + self.capture_closed = Some(closed_rx); + cx.background_spawn(async move { + active.cancel(); + closed_tx.send(()).ok(); + }) + .detach(); + } + + fn log_cancelled(&mut self, before: &Phase, now: Instant, outcome: &str) { + let Phase::Recording { mode, started, .. } = before else { + return; + }; + let settings = self.take_recording_settings(); + let mut log = recording_log_for(&settings, mode_label(*mode, now)); + log.audio_ms = now.saturating_duration_since(*started).as_millis(); + log.emit(outcome); + } + + fn take_recording_settings(&mut self) -> VoiceInputSettings { + self.recording_settings + .take() + .unwrap_or_else(|| self.settings.clone()) + } + + fn transcribe(&mut self, mode: Mode, cx: &mut Context) { + let settings = self.take_recording_settings(); + self.recording_log = Some(recording_log_for( + &settings, + mode_label(mode, Instant::now()), + )); + let stopping = self.stopping.take(); + let job = TranscribeJob { + settings, + http: self.http.clone(), + loopback_http: self.loopback_http.clone(), + models_dir: self.models_dir.clone(), + system_models_dir: self.system_models_dir.clone(), + hash_cache: self.hash_cache.clone(), + transcriber: self.transcriber.clone(), + }; + self.transcription = Some(cx.spawn(async move |this, cx| { + let recording = match stopping { + Some(stopping) => stopping.await, + None => Recording::default(), + }; + let audio_ms = audio_ms(&recording); + let started = Instant::now(); + let input = job.run(recording, cx).await; + let transcribe_ms = started.elapsed().as_millis(); + this.update(cx, |this, cx| { + if let Some(log) = &mut this.recording_log { + log.audio_ms = audio_ms; + log.transcribe_ms = transcribe_ms; + } + this.handle(input, cx); + }) + .ok(); + })); + } + + fn start_download(&mut self, cx: &mut Context) { + let spec = models::spec(self.settings.model); + self.download_spec = Some(spec); + let (cancel_tx, cancel_rx) = oneshot::channel(); + self.download_cancel = Some(cancel_tx); + let (progress_tx, progress_rx) = smol::channel::unbounded::(); + let progress = move |bytes: u64| { + // Closed only once the download has ended; a late figure is not needed then. + progress_tx.try_send(bytes).ok(); + }; + // `download_model` writes and syncs with blocking `std::fs`: background only. + let downloading = cx.background_spawn(download_model( + self.http.clone(), + spec, + self.models_dir.clone(), + progress, + cancel_rx, + )); + let forward = cx.spawn(async move |this, cx| { + while let Ok(bytes) = progress_rx.recv().await { + let delivered = this.update(cx, |this, cx| { + this.handle(Input::DownloadProgress(bytes), cx) + }); + if delivered.is_err() { + break; + } + } + }); + let finish = cx.spawn(async move |this, cx| { + let result = downloading.await; + this.update(cx, |this, cx| { + this.download_cancel = None; + match result { + Ok(_) => this.handle(Input::DownloadDone, cx), + // Cancelled by the user: the machine is already idle (R6.6). + Err(DownloadError::Cancelled) => {} + Err(error) => { + this.handle(Input::DownloadFailed(download_message(spec, &error)), cx) + } + } + }) + .ok(); + }); + self.download = Some((forward, finish)); + } + + /// Re-renders every second while recording, for the elapsed time. + fn sync_ticker(&mut self, cx: &mut Context) { + let recording = matches!(self.machine.phase(), Phase::Recording { .. }); + if !recording { + self.ticker = None; + } else if self.ticker.is_none() { + self.ticker = Some(cx.spawn(async move |this, cx| { + loop { + cx.background_executor().timer(TICK).await; + let ticked = this.update(cx, |this, cx| { + if this.recording_owner_closed() { + // Abandoned from a task of its own: the cancel drops this ticker, + // which must not happen from inside the ticker's own poll. + cx.spawn(async move |this, cx| { + this.update(cx, |this, cx| { + this.abandon_closed_owner(None, cx); + }) + .ok(); + }) + .detach(); + } else { + cx.emit(VoiceEvent::Changed); + } + }); + if ticked.is_err() { + break; + } + } + })); + } + } +} + +/// Everything a transcription needs, captured when it starts so a settings change mid-flight +/// does not mix two configurations. +struct TranscribeJob { + settings: VoiceInputSettings, + http: Arc, + loopback_http: Option>, + models_dir: PathBuf, + system_models_dir: PathBuf, + hash_cache: Arc, + transcriber: Arc>, +} + +impl TranscribeJob { + /// Turns the recording into the input the machine expects: `Transcribed` (empty when nothing + /// was said) or `TranscribeFailed` with the user-facing message. + async fn run(self, recording: Recording, cx: &mut AsyncApp) -> Input { + let pcm = cx + .background_spawn(async move { + audio_prep::to_whisper_pcm(&recording.samples, recording.sample_rate) + }) + .await; + if audio_prep::speechless(&pcm) { + // Too short or too quiet: nothing is sent anywhere (R4.6). + return Input::Transcribed { + text: String::new(), + }; + } + if self.settings.remote.enabled { + self.remote(pcm, cx).await + } else { + self.local(pcm, cx).await + } + } + + async fn local(self, pcm: Vec, cx: &mut AsyncApp) -> Input { + if let Err(level) = local::cpu_supported() { + return Input::TranscribeFailed(format!("Voice input needs a CPU with {level}")); + } + let spec = models::spec(self.settings.model); + let language = self.settings.language; + let models_dir = self.models_dir; + let system_dir = self.system_models_dir; + let cache = self.hash_cache; + let transcriber = self.transcriber; + cx.background_spawn(async move { + let Some(resolved) = models::resolve_model_in(spec, &system_dir, &models_dir, &cache) + else { + return Input::TranscribeFailed(format!( + "The speech model {} is missing; click the microphone to download it again", + model_name(spec.model) + )); + }; + let result = transcriber + .lock() + .unwrap_or_else(PoisonError::into_inner) + .transcribe(spec.model, &resolved.path, &language, &pcm); + match result { + Ok(text) => Input::Transcribed { text }, + Err(error @ LocalError::Unreadable { .. }) => { + Input::TranscribeFailed(unreadable_model(error, &resolved, spec)) + } + Err(error) => Input::TranscribeFailed(format!("{error:#}")), + } + }) + .await + } + + async fn remote(self, pcm: Vec, cx: &mut AsyncApp) -> Input { + let remote = self.settings.remote; + let Some(raw) = remote.url.as_deref() else { + return Input::TranscribeFailed( + "The remote voice URL is not set (voice_input.remote.url)".to_string(), + ); + }; + let url = match remote::validate_url(raw) { + Ok(url) => url, + // Refused before a byte is sent (R8.4). + Err(error) => return Input::TranscribeFailed(format!("{error:#}")), + }; + // `validate_url` admits plain http to loopback hosts only: with a proxy-less client it goes + // direct (R8.9), which ignores the environment's proxies too. + let (http, refusal) = match self.loopback_http { + Some(loopback) if url.scheme() == "http" => { + let refusal = remote::proxy_refusal(&url, loopback.proxy(), &[const { None }; 6]); + (loopback, refusal) + } + _ => { + let proxy_env = remote::PROXY_ENV_VARS.map(|name| std::env::var(name).ok()); + let refusal = remote::proxy_refusal(&url, self.http.proxy(), &proxy_env); + (self.http, refusal) + } + }; + if let Some(refusal) = refusal { + return Input::TranscribeFailed(refusal); + } + // Read per request and never stored. An empty name and an empty value are both no key + // (R8.5, R8.8). + let key = Some(remote.api_key_env.as_str()) + .filter(|name| !name.is_empty()) + .and_then(|name| std::env::var(name).ok()) + .filter(|key| !key.is_empty()); + let language = self.settings.language; + let model = remote.model; + let (body, boundary) = cx + .background_spawn(async move { + let wav = audio_prep::wav_pcm16(&pcm); + let boundary = remote::new_boundary(&wav); + let body = remote::multipart(&boundary, &wav, &model, &language); + (body, boundary) + }) + .await; + // Awaited here rather than on the background executor: the request is non-blocking I/O, + // and `remote::transcribe`'s future is not `Send` (its log guard uses `Cell`). + let result = remote::transcribe(http, &url, key, body, &boundary).await; + match result { + Ok(text) => Input::Transcribed { text }, + // No local fallback, in either direction (R8.6). + Err(error) => Input::TranscribeFailed(format!("{error:#}")), + } + } +} + +/// Handles a model whisper.cpp could not load (R7.3). A downloaded model whose bytes no longer +/// hash to the pinned value is deleted so the next use downloads it again; a model that still +/// hashes right failed to load for another reason and is kept, and a packaged one is never +/// touched (it belongs to the package manager). Runs on a background thread: it may hash. +fn unreadable_model(error: LocalError, resolved: &ResolvedModel, spec: &ModelSpec) -> String { + let LocalError::Unreadable { source, .. } = &error else { + return format!("{error:#}"); + }; + let name = model_name(spec.model); + let path = resolved.path.display().to_string(); + // Kept: the message names the failure without promising a download that will not happen. + let keep = |reason: &str| { + log::warn!( + "voice_input event=model_unreadable model={name} action=kept reason={reason} path={path:?}" + ); + format!("The speech model was unreadable: {source:#}") + }; + if resolved.origin == ModelOrigin::System { + return keep("system_package"); + } + match models::matches_pinned_hash(&resolved.path, spec) { + Ok(true) => return keep("checksum_matches"), + Ok(false) => {} + Err(hash_error) if hash_error.kind() == io::ErrorKind::NotFound => {} + Err(hash_error) => { + log::warn!( + "voice_input event=model_hash_failed model={name} path={path:?} error={:?}", + hash_error.to_string() + ); + } + } + match fs::remove_file(&resolved.path) { + Ok(()) => { + log::warn!( + "voice_input event=model_unreadable model={name} action=deleted path={path:?}" + ); + format!("{error:#}") + } + Err(remove_error) if remove_error.kind() == io::ErrorKind::NotFound => format!("{error:#}"), + Err(remove_error) => { + log::warn!( + "voice_input event=model_delete_failed model={name} path={path:?} error={:?}", + remove_error.to_string() + ); + format!( + "The speech model was unreadable and could not be deleted ({path}): {remove_error}" + ) + } + } +} + +/// The host audio goes to under `settings`, or `None` in local mode (R8.3). +fn settings_remote_host(settings: &VoiceInputSettings) -> Option { + if !settings.remote.enabled { + return None; + } + let url = url::Url::parse(settings.remote.url.as_deref()?).ok()?; + url.host_str().map(str::to_string) +} + +fn recording_log_for(settings: &VoiceInputSettings, mode: &'static str) -> RecordingLog { + let remote = &settings.remote; + if remote.enabled { + let host = remote + .url + .as_deref() + .and_then(|raw| url::Url::parse(raw).ok()) + .map(|url| redacted_host(&url)) + .unwrap_or_else(|| "none".to_string()); + RecordingLog { + mode, + backend: "remote", + model: format!("{:?}", remote.model), + host: Some(host), + audio_ms: 0, + transcribe_ms: 0, + } + } else { + RecordingLog { + mode, + backend: "local", + model: model_name(settings.model).to_string(), + host: None, + audio_ms: 0, + transcribe_ms: 0, + } + } +} + +/// The R6.5/R6.6 message for a failed download, naming the model. +fn download_message(spec: &ModelSpec, error: &DownloadError) -> String { + let name = model_name(spec.model); + match error { + DownloadError::Mismatch { .. } => { + format!("The speech model {name} did not match its checksum and was deleted") + } + DownloadError::Http(source) | DownloadError::Io(source) => { + format!("Could not download the speech model {name}: {source:#}") + } + DownloadError::Cancelled => { + format!("The download of the speech model {name} was cancelled") + } + } +} + +/// `toggle` or `hold`; a press still pending is named by what it would become now. +fn mode_label(mode: Mode, now: Instant) -> &'static str { + match mode { + Mode::Toggle => "toggle", + Mode::Hold => "hold", + Mode::Pending(pressed_at) if now.saturating_duration_since(pressed_at) <= TAP_WINDOW => { + "toggle" + } + Mode::Pending(_) => "hold", + } +} + +fn phase_name(phase: &Phase) -> &'static str { + match phase { + Phase::Idle => "idle", + Phase::AwaitingConsent { .. } => "awaiting_consent", + Phase::Downloading { .. } => "downloading", + Phase::Starting { .. } => "starting", + Phase::Recording { .. } => "recording", + Phase::Transcribing { .. } => "transcribing", + } +} + +fn audio_ms(recording: &Recording) -> u128 { + if recording.sample_rate == 0 { + return 0; + } + recording.samples.len() as u128 * 1000 / u128::from(recording.sample_rate) +} + +/// Host and port only: never the user-info, path or query of the configured URL (R10.3). +fn redacted_host(url: &url::Url) -> String { + let host = url.host_str().unwrap_or("none"); + match url.port() { + Some(port) => format!("{host}:{port}"), + None => host.to_string(), + } +} + +#[cfg(test)] +mod audio_prep_tests; +#[cfg(test)] +mod capture_formats_tests; +#[cfg(test)] +mod capture_i24_sign_tests; +#[cfg(test)] +mod capture_tests; +#[cfg(test)] +mod deliver_tests; +#[cfg(test)] +mod download_owner_tests; +#[cfg(test)] +mod entity_hardening_tests; +#[cfg(test)] +mod entity_tests; +#[cfg(test)] +mod local_lang_tests; +#[cfg(test)] +mod local_tests; +#[cfg(test)] +mod models_system_tests; +#[cfg(test)] +mod models_tests; +#[cfg(test)] +mod remote_host_tests; +#[cfg(test)] +mod remote_loopback_tests; +#[cfg(test)] +mod remote_redirect_tests; +#[cfg(test)] +mod remote_tests; +#[cfg(test)] +mod settings_tests; +#[cfg(test)] +mod state_release_tests; +#[cfg(test)] +mod state_stop_tests; +#[cfg(test)] +mod state_tests; +#[cfg(test)] +mod transcript_tests; diff --git a/crates/voice_input/tests/download.rs b/crates/voice_input/tests/download.rs new file mode 100644 index 0000000..903e831 --- /dev/null +++ b/crates/voice_input/tests/download.rs @@ -0,0 +1,496 @@ +// Story 044, task 3.1 — model download with progress, cancel and verification (R6.2, R6.4, R6.5, +// R6.6, R6.9, R10.2; Q17). Integration test: `cargo test -p voice_input --test download`. +// +// Contract (design.md C1.3): `voice_input::download::download_model(http: Arc, +// spec: &'static ModelSpec, dir: PathBuf, progress: impl Fn(u64) + Send + 'static, +// cancel: oneshot::Receiver<()>) -> Result`, with +// `DownloadError::{Cancelled, Http(_), Io(_), Mismatch { model, expected, got, bytes }}` and a +// `Display` naming the model. The URL is `models::url(spec)`. Assumed public paths — adjust the +// `use` only. Dev-dependencies: `http_client/test-support`, `tempfile`, `sha2` (normal dep). +// +// Driven with `futures::executor::block_on`: `download_model` takes no executor, so nothing here +// depends on gpui's scheduler. A watchdog thread turns a hang into a failure. +// +// Hostile halves: the exact body is accepted AND the same size with one byte changed is refused; +// one byte short AND one byte long are refused; an oversize body is aborted as soon as it passes the +// pinned size (a body that never ends must not hang the download); a cancel halfway AND a future +// dropped halfway (the "kill") leave neither the final file nor a staging file (Q17); redirects are +// followed (Hugging Face answers 302 to its CDN). + +use std::{ + path::Path, + pin::Pin, + sync::{Arc, Mutex}, + task::{Context, Poll}, + thread, + time::Duration, +}; + +use futures::{AsyncRead, FutureExt as _, channel::oneshot, future::Either}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Method, RedirectPolicy, Response}; +use sha2::{Digest as _, Sha256}; +use voice_input::download::{DownloadError, download_model}; +use voice_input::models::{ModelSpec, VoiceModel, url}; + +const FILE: &str = "ggml-test.bin"; + +fn body(len: usize) -> Vec { + (0..len).map(|i| (i * 31 % 251) as u8).collect() +} + +fn sha_hex(bytes: &[u8]) -> String { + Sha256::digest(bytes) + .iter() + .map(|b| format!("{b:02x}")) + .collect() +} + +fn test_spec(bytes: &[u8]) -> &'static ModelSpec { + Box::leak(Box::new(ModelSpec { + model: VoiceModel::Tiny, + file: FILE, + size: bytes.len() as u64, + sha256: Box::leak(sha_hex(bytes).into_boxed_str()), + })) +} + +#[derive(Clone, Debug)] +struct Seen { + uri: String, + method: Method, + redirects: Option, +} + +type Log = Arc>>; + +fn client( + status: u16, + make_body: impl Fn() -> AsyncBody + Send + Sync + 'static, +) -> (Arc, Log) { + let log: Log = Arc::default(); + let seen = log.clone(); + let make_body = Arc::new(make_body); + let http = FakeHttpClient::create(move |req| { + seen.lock().expect("lock").push(Seen { + uri: req.uri().to_string(), + method: req.method().clone(), + redirects: req.extensions().get::().cloned(), + }); + let make_body = make_body.clone(); + async move { Ok(Response::builder().status(status).body(make_body())?) } + }); + (http, log) +} + +fn serving(bytes: Vec) -> (Arc, Log) { + client(200, move || AsyncBody::from(bytes.clone())) +} + +/// Yields `data`, then pends forever (never EOF) after signalling `reached`. +struct Stalling { + data: Vec, + pos: usize, + reached: Option>, +} + +impl AsyncRead for Stalling { + fn poll_read( + mut self: Pin<&mut Self>, + _cx: &mut Context<'_>, + buf: &mut [u8], + ) -> Poll> { + if self.pos < self.data.len() { + let n = buf.len().min(self.data.len() - self.pos).min(4096); + let start = self.pos; + buf[..n].copy_from_slice(&self.data[start..start + n]); + self.pos += n; + return Poll::Ready(Ok(n)); + } + if let Some(tx) = self.reached.take() { + let _ = tx.send(()); + } + Poll::Pending + } +} + +fn stalling_client(data: Vec) -> (Arc, Log, oneshot::Receiver<()>) { + let (tx, rx) = oneshot::channel(); + let tx = Arc::new(Mutex::new(Some(tx))); + let (http, log) = client(200, move || { + AsyncBody::from_reader(Stalling { + data: data.clone(), + pos: 0, + reached: tx.lock().expect("lock").take(), + }) + }); + (http, log, rx) +} + +fn block_on_with_watchdog(fut: impl std::future::Future) -> T { + let (tx, rx) = oneshot::channel::<()>(); + thread::spawn(move || { + thread::sleep(Duration::from_secs(20)); + let _ = tx.send(()); + }); + match futures::executor::block_on(futures::future::select(Box::pin(fut), rx)) { + Either::Left((out, _)) => out, + Either::Right(_) => panic!("the download hung for 20 s"), + } +} + +fn leftovers(dir: &Path) -> Vec { + match std::fs::read_dir(dir) { + Ok(entries) => entries + .filter_map(|e| e.ok()) + .map(|e| e.file_name().to_string_lossy().into_owned()) + .collect(), + Err(_) => Vec::new(), + } +} + +fn never_cancelled() -> oneshot::Receiver<()> { + let (tx, rx) = oneshot::channel(); + std::mem::forget(tx); + rx +} + +#[test] +fn download_success_verifies_and_renames_into_place() { + let bytes = body(300_000); + let spec = test_spec(&bytes); + let (http, log) = serving(bytes.clone()); + let tmp = tempfile::tempdir().expect("tempdir"); + let dir = tmp.path().join("voice").join("models"); // created by the download + let progress: Arc>> = Arc::default(); + let p = progress.clone(); + let path = block_on_with_watchdog(download_model( + http, + spec, + dir.clone(), + move |n| p.lock().expect("lock").push(n), + never_cancelled(), + )) + .expect("the pinned bytes are accepted"); + + assert_eq!(path, dir.join(FILE)); + assert_eq!(std::fs::read(&path).expect("final file"), bytes); + assert_eq!( + leftovers(&dir), + vec![FILE.to_string()], + "no staging file left" + ); + let seen = log.lock().expect("lock").clone(); + assert_eq!(seen.len(), 1, "one request"); + assert_eq!(seen[0].uri, url(spec), "the pinned URL"); + assert_eq!(seen[0].method, Method::GET); + assert_eq!( + seen[0].redirects, + Some(RedirectPolicy::FollowAll), + "the CDN redirect is followed" + ); + let progress = progress.lock().expect("lock").clone(); + assert!( + progress.windows(2).all(|w| w[0] <= w[1]), + "monotonic: {progress:?}" + ); + assert!( + progress.iter().all(|n| *n <= spec.size), + "never past the pinned size: {progress:?}" + ); +} + +#[test] +fn download_same_size_wrong_digest_is_a_mismatch_and_deleted() { + let bytes = body(100_000); + let spec = test_spec(&bytes); + let mut tampered = bytes.clone(); + tampered[50_000] ^= 0xff; + let (http, _) = serving(tampered); + let tmp = tempfile::tempdir().expect("tempdir"); + let err = block_on_with_watchdog(download_model( + http, + spec, + tmp.path().into(), + |_| {}, + never_cancelled(), + )) + .expect_err("a changed byte is refused"); + match &err { + DownloadError::Mismatch { + model, + expected, + got, + bytes, + } => { + assert_eq!(*model, VoiceModel::Tiny); + assert_eq!(*expected, spec.sha256); + assert_ne!(got.as_str(), spec.sha256); + assert_eq!(*bytes, spec.size); + } + other => panic!("expected Mismatch, got {other:?}"), + } + let shown = err.to_string(); + assert!( + shown.contains("tiny"), + "R6.5 the error names the model: {shown}" + ); + assert!( + leftovers(tmp.path()).is_empty(), + "R6.5 deleted: {:?}", + leftovers(tmp.path()) + ); +} + +#[test] +fn download_one_byte_short_or_long_is_refused() { + let bytes = body(50_000); + let spec = test_spec(&bytes); + for served in [ + bytes[..bytes.len() - 1].to_vec(), + [bytes.clone(), vec![0]].concat(), + ] { + let (http, _) = serving(served.clone()); + let tmp = tempfile::tempdir().expect("tempdir"); + let result = block_on_with_watchdog(download_model( + http, + spec, + tmp.path().into(), + |_| {}, + never_cancelled(), + )); + assert!( + matches!(result, Err(DownloadError::Mismatch { .. })), + "{} bytes: {result:?}", + served.len() + ); + assert!( + leftovers(tmp.path()).is_empty(), + "{:?}", + leftovers(tmp.path()) + ); + } +} + +#[test] +fn download_oversize_aborts_without_waiting_for_the_end() { + let bytes = body(40_000); + let spec = test_spec(&bytes); + // spec.size + 1 bytes, then a body that never ends: only an early abort returns. + let (http, _, _reached) = stalling_client(body(40_001)); + let tmp = tempfile::tempdir().expect("tempdir"); + let result = block_on_with_watchdog(download_model( + http, + spec, + tmp.path().into(), + |_| {}, + never_cancelled(), + )); + assert!( + matches!(result, Err(DownloadError::Mismatch { .. })), + "{result:?}" + ); + assert!( + leftovers(tmp.path()).is_empty(), + "{:?}", + leftovers(tmp.path()) + ); +} + +#[test] +fn download_cancel_halfway_leaves_nothing() { + let bytes = body(200_000); + let spec = test_spec(&bytes); + let (http, _, reached) = stalling_client(bytes[..100_000].to_vec()); + let tmp = tempfile::tempdir().expect("tempdir"); + let (cancel_tx, cancel_rx) = oneshot::channel(); + let dir = tmp.path().to_path_buf(); + let result = block_on_with_watchdog(async move { + let download = download_model(http, spec, dir, |_| {}, cancel_rx).fuse(); + let trigger = async move { + let _ = reached.await; + let _ = cancel_tx.send(()); + futures::future::pending::<()>().await; + } + .fuse(); + futures::pin_mut!(download, trigger); + futures::select! { + r = download => r, + _ = trigger => unreachable!(), + } + }); + assert!( + matches!(result, Err(DownloadError::Cancelled)), + "{result:?}" + ); + assert!( + !tmp.path().join(FILE).exists(), + "Q17 no file at the final path" + ); + assert!( + leftovers(tmp.path()).is_empty(), + "R6.6 partial file deleted: {:?}", + leftovers(tmp.path()) + ); +} + +#[test] +fn download_dropped_halfway_leaves_nothing() { + let bytes = body(200_000); + let spec = test_spec(&bytes); + let (http, _, reached) = stalling_client(bytes[..100_000].to_vec()); + let tmp = tempfile::tempdir().expect("tempdir"); + let dir = tmp.path().to_path_buf(); + block_on_with_watchdog(async move { + let download = download_model(http, spec, dir, |_| {}, never_cancelled()).fuse(); + futures::pin_mut!(download); + let mut reached = reached.fuse(); + futures::select! { + r = download => panic!("a stalled body cannot finish: {r:?}"), + _ = reached => {} + } + // `download` is dropped here, halfway: the "kill". + }); + assert!( + !tmp.path().join(FILE).exists(), + "Q17 no file at the final path" + ); + assert!( + leftovers(tmp.path()).is_empty(), + "the drop guard removed the staging file: {:?}", + leftovers(tmp.path()) + ); +} + +#[test] +fn download_http_404_is_an_http_error_and_writes_nothing() { + let bytes = body(1_000); + let spec = test_spec(&bytes); + let (http, _) = client(404, || AsyncBody::from("Not Found")); + let tmp = tempfile::tempdir().expect("tempdir"); + let result = block_on_with_watchdog(download_model( + http, + spec, + tmp.path().into(), + |_| {}, + never_cancelled(), + )); + match result { + Err(DownloadError::Http(e)) => assert!(format!("{e:#}").contains("404"), "{e:#}"), + other => panic!("expected Http, got {other:?}"), + } + assert!( + leftovers(tmp.path()).is_empty(), + "{:?}", + leftovers(tmp.path()) + ); +} + +#[test] +fn download_transport_error_is_an_http_error() { + let bytes = body(1_000); + let spec = test_spec(&bytes); + let http: Arc = + FakeHttpClient::create(|_| async move { Err(anyhow::anyhow!("connection refused")) }); + let tmp = tempfile::tempdir().expect("tempdir"); + let result = block_on_with_watchdog(download_model( + http, + spec, + tmp.path().into(), + |_| {}, + never_cancelled(), + )); + assert!(matches!(result, Err(DownloadError::Http(_))), "{result:?}"); + assert!(leftovers(tmp.path()).is_empty()); +} + +// R10.2: one key=value line per download, with model, bytes and outcome. +mod log_capture { + use std::sync::{Mutex, OnceLock}; + use std::thread::ThreadId; + + pub struct Capture(pub Mutex>); + + impl log::Log for Capture { + fn enabled(&self, _: &log::Metadata) -> bool { + true + } + fn log(&self, record: &log::Record) { + self.0 + .lock() + .expect("lock") + .push((std::thread::current().id(), record.args().to_string())); + } + fn flush(&self) {} + } + + pub fn install() -> &'static Capture { + static CAPTURE: OnceLock<&'static Capture> = OnceLock::new(); + CAPTURE.get_or_init(|| { + let capture: &'static Capture = Box::leak(Box::new(Capture(Mutex::new(Vec::new())))); + log::set_logger(capture).expect("no other logger in this test binary"); + log::set_max_level(log::LevelFilter::Trace); + capture + }) + } + + pub fn lines_of_this_thread(capture: &Capture) -> Vec { + let me = std::thread::current().id(); + capture + .0 + .lock() + .expect("lock") + .iter() + .filter(|(t, _)| *t == me) + .map(|(_, l)| l.clone()) + .collect() + } +} + +#[test] +fn download_logs_one_key_value_line_per_outcome() { + let capture = log_capture::install(); + let bytes = body(10_000); + let spec = test_spec(&bytes); + let tmp = tempfile::tempdir().expect("tempdir"); + + let (http, _) = serving(bytes.clone()); + block_on_with_watchdog(download_model( + http, + spec, + tmp.path().join("a"), + |_| {}, + never_cancelled(), + )) + .expect("ok"); + let (http, _) = serving(body(9_999)); + let _ = block_on_with_watchdog(download_model( + http, + spec, + tmp.path().join("b"), + |_| {}, + never_cancelled(), + )); + + let lines: Vec = log_capture::lines_of_this_thread(capture) + .into_iter() + .filter(|l| { + l.starts_with("voice_input event=") + && l.contains("model=tiny") + && l.contains("outcome=") + }) + .collect(); + assert_eq!(lines.len(), 2, "one line per download: {lines:?}"); + assert!(lines[0].contains("bytes=10000"), "{}", lines[0]); + assert!(lines[1].contains("bytes=9999"), "{}", lines[1]); + assert_ne!( + lines[0].split("outcome=").nth(1), + lines[1].split("outcome=").nth(1), + "success and mismatch are distinguishable" + ); + for line in &lines { + assert!(line.contains("duration"), "carries the duration: {line}"); + for pair in line.split_whitespace().skip(1) { + assert!(pair.contains('='), "key=value only: {line}"); + } + } +} diff --git a/crates/voice_input/tests/download_hygiene.rs b/crates/voice_input/tests/download_hygiene.rs new file mode 100644 index 0000000..429757f --- /dev/null +++ b/crates/voice_input/tests/download_hygiene.rs @@ -0,0 +1,360 @@ +// Story 044, task 8.1 (refine v3) — download hygiene: a stalled header wait and stale staging files +// (R6.6, R6.12, R6.13; design.md C1.3, C1.10). Run: `cargo test -p voice_input --test +// download_hygiene`. +// +// Contract: `download::download_model_with_stall(http, spec, dir, progress, cancel, stall)` fails +// with `DownloadError::Http` when the response headers do not arrive within `stall`, and with +// `DownloadError::Cancelled` when `cancel` fires while it waits for them; either way the models +// directory is left empty. `download::sweep_staging(dir, older_than) -> usize` deletes the regular +// files named `.download-*` whose last write is older than `older_than`, never follows a symlink, +// and returns how many it deleted. `voice_input::init_in` sweeps the user models directory once, +// with 60 s, and never `/system`. +// +// Hostile halves, the wrong collapse first: a sweep that matches too much — a model file, a name +// that only contains `.download-`, a directory or a symlink named `.download-*`, a staging file +// written a moment ago by another running Zeo, anything in the system directory — keeps every one +// of them; then the wrong split — two stale staging files of different models are both deleted +// and counted; then the plain case. For the header wait: headers that arrive late but inside the +// stall are not aborted, before headers that never arrive are. + +use std::{ + fs, + path::Path, + sync::{ + Arc, + atomic::{AtomicUsize, Ordering}, + }, + thread, + time::{Duration, SystemTime}, +}; + +use futures::{ + channel::oneshot, + future::{self, Either}, +}; +use gpui::TestAppContext; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use settings::SettingsStore; +use voice_input::capture::{ActiveCapture, CaptureBackend, CaptureEvent}; +use voice_input::download::{DownloadError, download_model_with_stall, sweep_staging}; +use voice_input::models::{ModelSpec, VoiceModel, url}; + +static SPEC: ModelSpec = ModelSpec { + model: VoiceModel::Tiny, + file: "ggml-hygiene.bin", + size: 1_000, + sha256: "0000000000000000000000000000000000000000000000000000000000000000", +}; + +/// Runs `download` for at most `limit`; `None` when it was still waiting then. +fn within(limit: Duration, download: impl Future) -> Option { + let (watchdog_tx, watchdog_rx) = oneshot::channel::<()>(); + thread::spawn(move || { + thread::sleep(limit); + watchdog_tx.send(()).ok(); + }); + match futures::executor::block_on(future::select(Box::pin(download), watchdog_rx)) { + Either::Left((result, _)) => Some(result), + Either::Right(_) => None, + } +} + +/// An HTTP client whose response headers never come. +fn headers_never_arrive(requests: Arc) -> Arc { + FakeHttpClient::create(move |_| { + requests.fetch_add(1, Ordering::SeqCst); + future::pending::>>() + }) +} + +fn entries(dir: &Path) -> Vec { + match fs::read_dir(dir) { + Ok(read) => read + .map(|entry| { + entry + .expect("dir entry") + .file_name() + .to_string_lossy() + .into_owned() + }) + .collect(), + Err(_) => Vec::new(), + } +} + +// ── R6.13: the wait for the response headers ─────────────────────────────── + +#[test] +fn download_hygiene_headers_arriving_inside_the_stall_are_not_aborted() { + let http: Arc = FakeHttpClient::create(|_| async { + // Answers after 300 ms; the stall below is 2 s. + let (ready_tx, ready_rx) = oneshot::channel::<()>(); + thread::spawn(move || { + thread::sleep(Duration::from_millis(300)); + ready_tx.send(()).ok(); + }); + ready_rx.await.ok(); + Ok(Response::builder() + .status(200) + .body(AsyncBody::from(vec![7u8; 10]))?) + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let (_cancel_tx, cancel_rx) = oneshot::channel(); + let result = within( + Duration::from_secs(10), + download_model_with_stall( + http, + &SPEC, + tmp.path().to_path_buf(), + |_| {}, + cancel_rx, + Duration::from_secs(2), + ), + ) + .expect("a late but timely answer finishes the download"); + assert!( + matches!(result, Err(DownloadError::Mismatch { bytes: 10, .. })), + "the body was read (10 wrong bytes): the header wait did not abort it: {result:?}" + ); +} + +#[test] +fn download_hygiene_headers_that_never_arrive_abort_as_http_within_the_stall() { + let requests = Arc::new(AtomicUsize::new(0)); + let tmp = tempfile::tempdir().expect("tempdir"); + let (_cancel_tx, cancel_rx) = oneshot::channel(); + let result = within( + Duration::from_secs(3), + download_model_with_stall( + headers_never_arrive(requests.clone()), + &SPEC, + tmp.path().to_path_buf(), + |_| {}, + cancel_rx, + Duration::from_millis(200), + ), + ); + assert!( + result.is_some(), + "R6.13 headers that never arrive hung the download past 3 s (stall: 200 ms)" + ); + match result { + Some(Err(DownloadError::Http(error))) => { + let shown = format!("{error:#}"); + assert!( + shown.contains(&url(&SPEC)), + "the error names the URL: {shown}" + ); + } + other => panic!("expected Http, got {other:?}"), + } + assert_eq!(requests.load(Ordering::SeqCst), 1, "one request, no retry"); + assert!( + entries(tmp.path()).is_empty(), + "R6.6 nothing left in the models dir: {:?}", + entries(tmp.path()) + ); +} + +#[test] +fn download_hygiene_cancel_during_the_header_wait_is_cancelled() { + let tmp = tempfile::tempdir().expect("tempdir"); + let (cancel_tx, cancel_rx) = oneshot::channel(); + thread::spawn(move || { + thread::sleep(Duration::from_millis(150)); + cancel_tx.send(()).ok(); + }); + let result = within( + Duration::from_secs(3), + download_model_with_stall( + headers_never_arrive(Arc::default()), + &SPEC, + tmp.path().to_path_buf(), + |_| {}, + cancel_rx, + // Far longer than the watchdog: only the cancel can end this wait. + Duration::from_secs(30), + ), + ); + assert!( + result.is_some(), + "R6.6 a cancel sent while the headers are awaited did not stop the download" + ); + assert!( + matches!(result, Some(Err(DownloadError::Cancelled))), + "expected Cancelled, got {result:?}" + ); + assert!( + entries(tmp.path()).is_empty(), + "nothing left: {:?}", + entries(tmp.path()) + ); +} + +// ── R6.12: stale staging files ───────────────────────────────────────────── + +const OLDER_THAN: Duration = Duration::from_secs(60); + +fn write_aged(path: &Path, age: Duration) { + fs::write(path, b"partial").expect("write"); + age_path(path, age); +} + +fn age_path(path: &Path, age: Duration) { + fs::File::open(path) + .expect("open") + .set_modified(SystemTime::now() - age) + .expect("set mtime"); +} + +#[test] +fn download_hygiene_sweep_keeps_everything_that_is_not_a_stale_staging_file() { + let tmp = tempfile::tempdir().expect("tempdir"); + let dir = tmp.path(); + let old = Duration::from_secs(600); + // The wrong collapse: every one of these is old, and none is a staging file to delete. + write_aged(&dir.join("ggml-base.bin"), old); + write_aged(&dir.join("x.download-ggml-base.bin-1"), old); + write_aged(&dir.join("download-ggml-base.bin-2"), old); + let staging_dir = dir.join(".download-d"); + fs::create_dir(&staging_dir).expect("dir"); + age_path(&staging_dir, old); + let outside = tempfile::tempdir().expect("tempdir"); + let target = outside.path().join("model.bin"); + write_aged(&target, old); + #[cfg(unix)] + std::os::unix::fs::symlink(&target, dir.join(".download-link")).expect("symlink"); + // Another running Zeo's live staging file: written a moment ago. + write_aged( + &dir.join(".download-ggml-tiny.bin-live"), + Duration::from_secs(5), + ); + // The one stale staging file. + write_aged(&dir.join(".download-ggml-small.bin-stale"), old); + + let removed = sweep_staging(dir, OLDER_THAN); + + let left = entries(dir); + assert!( + !left.contains(&".download-ggml-small.bin-stale".to_string()), + "R6.12 the stale staging file is deleted: {left:?}" + ); + assert_eq!(removed, 1, "exactly one file deleted: {left:?}"); + for kept in [ + "ggml-base.bin", + "x.download-ggml-base.bin-1", + "download-ggml-base.bin-2", + ".download-d", + ".download-ggml-tiny.bin-live", + ] { + assert!(left.contains(&kept.to_string()), "{kept} is kept: {left:?}"); + } + #[cfg(unix)] + { + assert!( + fs::symlink_metadata(dir.join(".download-link")).is_ok(), + "a symlink named .download-* is not a staging file: {left:?}" + ); + assert!(target.exists(), "a symlink's target is never touched"); + } + assert!( + staging_dir.is_dir(), + "a directory named .download-* is kept" + ); +} + +#[test] +fn download_hygiene_sweep_deletes_every_stale_staging_file_and_counts_them() { + let tmp = tempfile::tempdir().expect("tempdir"); + let dir = tmp.path(); + // The wrong split: different models, different random suffixes, both stale. + write_aged( + &dir.join(".download-ggml-tiny.bin-a1b2c3"), + Duration::from_secs(61 + 60), + ); + write_aged( + &dir.join(".download-ggml-base.bin-ZZ"), + Duration::from_secs(3_600), + ); + write_aged(&dir.join("ggml-tiny.bin"), Duration::from_secs(3_600)); + + let removed = sweep_staging(dir, OLDER_THAN); + + assert_eq!( + removed, + 2, + "R6.12 both stale staging files: {:?}", + entries(dir) + ); + assert_eq!(entries(dir), vec!["ggml-tiny.bin".to_string()]); +} + +#[test] +fn download_hygiene_sweep_of_a_missing_dir_returns_zero() { + let tmp = tempfile::tempdir().expect("tempdir"); + assert_eq!( + sweep_staging(&tmp.path().join("never-created"), OLDER_THAN), + 0 + ); +} + +// ── R6.12 at init: the user models dir, never the system one ─────────────── + +struct NoMicrophone; + +impl CaptureBackend for NoMicrophone { + fn start( + &self, + _device: Option, + _events: smol::channel::Sender, + ) -> anyhow::Result> { + anyhow::bail!("no microphone in this test") + } +} + +#[gpui::test] +async fn download_hygiene_init_sweeps_the_user_models_dir_only(cx: &mut TestAppContext) { + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let models = tmp.path().join("voice").join("models"); + let system = models.join("system"); + fs::create_dir_all(&system).expect("dirs"); + let old = Duration::from_secs(600); + write_aged(&models.join(".download-ggml-base.bin-stale"), old); + write_aged( + &models.join(".download-ggml-tiny.bin-live"), + Duration::from_secs(10), + ); + write_aged(&system.join(".download-ggml-base.bin-stale"), old); + + let http: Arc = + FakeHttpClient::create(|_| async { anyhow::bail!("no request is expected") }); + let capture: Arc = Arc::new(NoMicrophone); + cx.update(|cx| voice_input::init_in(http, models.clone(), capture, cx)); + cx.run_until_parked(); + + let deadline = std::time::Instant::now() + Duration::from_secs(5); + while models.join(".download-ggml-base.bin-stale").exists() + && std::time::Instant::now() < deadline + { + cx.run_until_parked(); + thread::sleep(Duration::from_millis(10)); + } + assert!( + !models.join(".download-ggml-base.bin-stale").exists(), + "R6.12 the stale staging file in the user models dir is deleted at init: {:?}", + entries(&models) + ); + assert!( + models.join(".download-ggml-tiny.bin-live").exists(), + "a staging file 10 s old belongs to a live download: kept" + ); + assert!( + system.join(".download-ggml-base.bin-stale").exists(), + "R6.12 the system model directory is never touched" + ); +} diff --git a/crates/voice_input/tests/download_hygiene_lock.rs b/crates/voice_input/tests/download_hygiene_lock.rs new file mode 100644 index 0000000..b474f2f --- /dev/null +++ b/crates/voice_input/tests/download_hygiene_lock.rs @@ -0,0 +1,162 @@ +// Story 044, task 8.1 fix round 1 (tech review L1): a staging file that is still being written is +// never swept, however old its last write looks. The age rule alone cannot tell: the final fsync +// and rename do not move the mtime, and a suspend moves the wall clock past the writer's monotonic +// stall timer. The download holds an exclusive lock on its staging file from creation until the +// rename, and the sweep keeps every file whose lock it cannot take. +// Run: `cargo test -p voice_input --test download_hygiene_lock`. + +use std::{ + fs, + path::{Path, PathBuf}, + pin::Pin, + sync::Arc, + task::{Context, Poll}, + thread, + time::{Duration, Instant, SystemTime}, +}; + +use futures::{AsyncRead, channel::oneshot}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use voice_input::download::{DownloadError, download_model_with_stall, sweep_staging}; +use voice_input::models::{ModelSpec, VoiceModel}; + +const OLDER_THAN: Duration = Duration::from_secs(60); +const OLD: Duration = Duration::from_secs(600); + +static SPEC: ModelSpec = ModelSpec { + model: VoiceModel::Tiny, + file: "ggml-lock.bin", + size: 1_000, + sha256: "0000000000000000000000000000000000000000000000000000000000000000", +}; + +fn age(path: &Path, by: Duration) { + fs::File::open(path) + .expect("open") + .set_modified(SystemTime::now() - by) + .expect("set mtime"); +} + +fn staging_files(dir: &Path) -> Vec { + fs::read_dir(dir) + .map(|read| { + read.filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| { + path.file_name() + .and_then(|name| name.to_str()) + .is_some_and(|name| name.starts_with(".download-")) + }) + .collect() + }) + .unwrap_or_default() +} + +#[test] +fn download_hygiene_lock_sweep_keeps_an_old_staging_file_someone_holds_locked() { + let tmp = tempfile::tempdir().expect("tempdir"); + let live = tmp.path().join(".download-ggml-small.bin-live"); + fs::write(&live, b"partial").expect("write"); + age(&live, OLD); + let holder = fs::OpenOptions::new() + .read(true) + .write(true) + .open(&live) + .expect("open"); + holder.lock().expect("lock"); + + let removed = sweep_staging(tmp.path(), OLDER_THAN); + + assert!( + live.exists(), + "a locked staging file belongs to a live download, whatever its mtime" + ); + assert_eq!(removed, 0, "nothing deleted"); + drop(holder); + assert_eq!( + sweep_staging(tmp.path(), OLDER_THAN), + 1, + "once the lock is released, the age rule applies again" + ); + assert!(!live.exists()); +} + +/// Ten bytes, then connected and silent until the test cancels. +struct TenBytesThenSilence { + sent: bool, +} + +impl AsyncRead for TenBytesThenSilence { + fn poll_read( + mut self: Pin<&mut Self>, + _: &mut Context<'_>, + buffer: &mut [u8], + ) -> Poll> { + if self.sent { + return Poll::Pending; + } + self.sent = true; + let n = buffer.len().min(10); + buffer[..n].fill(7); + Poll::Ready(Ok(n)) + } +} + +#[test] +fn download_hygiene_lock_a_running_download_holds_its_staging_file() { + let http: Arc = FakeHttpClient::create(|_| async { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from_reader(TenBytesThenSilence { sent: false }))?) + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let dir = tmp.path().to_path_buf(); + let (cancel_tx, cancel_rx) = oneshot::channel(); + let download = thread::spawn({ + let dir = dir.clone(); + move || { + futures::executor::block_on(download_model_with_stall( + http, + &SPEC, + dir, + |_| {}, + cancel_rx, + Duration::from_secs(20), + )) + } + }); + + let deadline = Instant::now() + Duration::from_secs(5); + let staging = loop { + if let Some(path) = staging_files(&dir).pop() + && fs::metadata(&path).is_ok_and(|metadata| metadata.len() == 10) + { + break path; + } + assert!( + Instant::now() < deadline, + "the download never wrote its staging file" + ); + thread::sleep(Duration::from_millis(10)); + }; + // What a suspend, or a long final fsync, makes a live file look like to another Zeo. + age(&staging, OLD); + + let removed = sweep_staging(&dir, OLDER_THAN); + + assert!( + staging.exists(), + "R6.12 the sweep deleted the staging file of a download still running" + ); + assert_eq!(removed, 0, "nothing deleted"); + cancel_tx.send(()).expect("the download is still waiting"); + let result = download.join().expect("download thread"); + assert!( + matches!(result, Err(DownloadError::Cancelled)), + "expected Cancelled, got {result:?}" + ); + assert!( + staging_files(&dir).is_empty(), + "R6.6 nothing left after the cancel" + ); +} diff --git a/crates/voice_input/tests/download_stall.rs b/crates/voice_input/tests/download_stall.rs new file mode 100644 index 0000000..969e66c --- /dev/null +++ b/crates/voice_input/tests/download_stall.rs @@ -0,0 +1,84 @@ +// Story 044, task 3.1 fix round 1 (tech review M1): a body that stays connected but never yields a +// byte must fail with `Http` after the stall timeout, and leave no file behind. +// Run: `cargo test -p voice_input --test download_stall`. + +use std::{ + pin::Pin, + sync::Arc, + task::{Context, Poll}, + thread, + time::Duration, +}; + +use futures::{AsyncRead, channel::oneshot, future::Either}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use voice_input::download::{DownloadError, STALL_TIMEOUT, download_model_with_stall}; +use voice_input::models::{ModelSpec, VoiceModel}; + +/// Connected, never at EOF, never a byte. +struct Silent; + +impl AsyncRead for Silent { + fn poll_read( + self: Pin<&mut Self>, + _: &mut Context<'_>, + _: &mut [u8], + ) -> Poll> { + Poll::Pending + } +} + +static SPEC: ModelSpec = ModelSpec { + model: VoiceModel::Tiny, + file: "ggml-stall.bin", + size: 1_000, + sha256: "0000000000000000000000000000000000000000000000000000000000000000", +}; + +#[test] +fn download_stall_timeout_is_thirty_seconds() { + assert_eq!(STALL_TIMEOUT, Duration::from_secs(30)); +} + +#[test] +fn download_silent_body_is_aborted_as_http_and_leaves_nothing() { + let http: Arc = FakeHttpClient::create(|_| async move { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from_reader(Silent))?) + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let (_cancel_tx, cancel_rx) = oneshot::channel(); + let download = download_model_with_stall( + http, + &SPEC, + tmp.path().to_path_buf(), + |_| {}, + cancel_rx, + Duration::from_millis(200), + ); + + let (watchdog_tx, watchdog_rx) = oneshot::channel::<()>(); + thread::spawn(move || { + thread::sleep(Duration::from_secs(10)); + watchdog_tx.send(()).ok(); + }); + let result = + match futures::executor::block_on(futures::future::select(Box::pin(download), watchdog_rx)) + { + Either::Left((result, _)) => result, + Either::Right(_) => panic!("a silent body hung the download for 10 s"), + }; + + match result { + Err(DownloadError::Http(error)) => { + assert!( + format!("{error:#}").contains("no data received"), + "{error:#}" + ) + } + other => panic!("expected Http, got {other:?}"), + } + let leftovers: Vec<_> = std::fs::read_dir(tmp.path()).expect("read_dir").collect(); + assert!(leftovers.is_empty(), "{leftovers:?}"); +} diff --git a/crates/voice_input/tests/insert_report.rs b/crates/voice_input/tests/insert_report.rs new file mode 100644 index 0000000..eec8834 --- /dev/null +++ b/crates/voice_input/tests/insert_report.rs @@ -0,0 +1,322 @@ +// Story 044, task 8.7 (refine v3) — a refused insert is logged as refused (R4.7, R10.1, R10.3, Q14; +// design.md C1.10 `report_insert_refused`). Run: `cargo test -p voice_input --test insert_report` +// (the filter `insert_report` matches too). +// +// An integration test, not `src/insert_report_tests.rs`: it reads the R10.1 line through a +// process-wide logger, and the lib test binary already installs one (`entity_tests.rs`'s +// `log_capture::install`, which panics when another is set). Its own binary keeps the two apart. +// +// Contract: after an `Insert` is emitted for `owner`, the recording's R10.1 line is pending until +// the effects flush. `VoiceInput::report_insert_refused(&mut self, owner: Owner)` called in that +// window (the thread view calls it from its `Insert` handler when the composer refuses) logs it +// with `outcome=refused`; nobody reporting logs it with `outcome=inserted`. A report for another +// owner, or with nothing pending, logs nothing. No line carries the transcript. +// +// Hostile halves, the wrong collapse first: a report for another composer does not turn this +// composer's line into `refused`, and a report with nothing pending invents no line; then the +// wrong split: the refused line is the recording's one line, not a second line beside an +// `inserted` one; then the plain `inserted` case. + +use std::{ + sync::{Arc, Mutex}, + time::{Duration, Instant}, +}; + +use gpui::{AppContext as _, Entity, EntityId, TestAppContext, UpdateGlobal as _}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use settings::SettingsStore; +use voice_input::capture::{ActiveCapture, CaptureBackend, CaptureEvent, Ended, Recording}; +use voice_input::state::{Input, Owner}; +use voice_input::{ViewState, VoiceEvent, VoiceInput, init_in}; + +const TRANSCRIPT: &str = "Confidential dictation words"; + +mod log_capture { + use std::sync::{Mutex, OnceLock}; + use std::thread::ThreadId; + + pub struct Capture(pub Mutex>); + + impl log::Log for Capture { + fn enabled(&self, _: &log::Metadata) -> bool { + true + } + fn log(&self, record: &log::Record) { + self.0 + .lock() + .expect("lock") + .push((std::thread::current().id(), record.args().to_string())); + } + fn flush(&self) {} + } + + pub fn install() -> &'static Capture { + static CAPTURE: OnceLock<&'static Capture> = OnceLock::new(); + CAPTURE.get_or_init(|| { + let capture: &'static Capture = Box::leak(Box::new(Capture(Mutex::new(Vec::new())))); + log::set_logger(capture).expect("no other logger in this test binary"); + log::set_max_level(log::LevelFilter::Trace); + capture + }) + } + + /// The `voice_input` lines logged on this test's thread (the gpui foreground). + pub fn mine(capture: &Capture) -> Vec { + let me = std::thread::current().id(); + capture + .0 + .lock() + .expect("lock") + .iter() + .filter(|(thread, line)| *thread == me && line.starts_with("voice_input ")) + .map(|(_, line)| line.clone()) + .collect() + } +} + +struct Mic; +struct Active; + +impl CaptureBackend for Mic { + fn start( + &self, + _device: Option, + _events: smol::channel::Sender, + ) -> anyhow::Result> { + Ok(Box::new(Active)) + } +} + +impl ActiveCapture for Active { + fn stop(self: Box) -> Recording { + let samples = (0..48_000).map(|i| 0.2 * (i as f32 * 0.05).sin()).collect(); + Recording { + samples, + sample_rate: 48_000, + ended: Ended::User, + } + } + fn cancel(self: Box) {} +} + +const REMOTE: &str = r#"{ "voice_input": { "remote": { "enabled": true, + "url": "http://127.0.0.1:9/v1/audio/transcriptions", "api_key_env": "ZEO_VOICE_TEST_UNSET_044_IR" } } }"#; + +/// Which owner, if any, the `Insert` handler reports as refused. +type Reporter = Arc>>; + +struct Harness { + vi: Entity, + capture: &'static log_capture::Capture, + inserts: Arc>>, + reporter: Reporter, + _tmp: tempfile::TempDir, +} + +impl Harness { + fn recording_lines(&self) -> Vec { + log_capture::mine(self.capture) + .into_iter() + .filter(|line| line.starts_with("voice_input event=recording ")) + .collect() + } + + fn record_until_insert(&self, owner: EntityId, cx: &mut TestAppContext) { + let input = || Input::Click { + owner: Owner(owner.as_u64()), + }; + self.vi.update(cx, |vi, cx| vi.handle(input(), cx)); + wait_until(cx, "recording", |cx| { + matches!( + self.vi.read_with(cx, |vi, _| vi.view_state(owner)), + ViewState::Recording { .. } + ) + }); + self.vi.update(cx, |vi, cx| vi.handle(input(), cx)); + wait_until(cx, "the insert", |_| { + self.inserts.lock().expect("lock").contains(&owner) + }); + // Lets the deferred fallback run. + cx.run_until_parked(); + } +} + +fn setup(cx: &mut TestAppContext) -> Harness { + let capture = log_capture::install(); + cx.executor().allow_parking(); + cx.update(|cx| { + let store = SettingsStore::test(cx); + cx.set_global(store); + SettingsStore::update_global(cx, |store, cx| { + store + .set_user_settings(REMOTE, cx) + .result() + .expect("settings parse"); + }); + }); + let tmp = tempfile::tempdir().expect("tempdir"); + let models = tmp.path().join("voice").join("models"); + let http: Arc = FakeHttpClient::create(|_| async { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from(format!(r#"{{"text": "{TRANSCRIPT}"}}"#)))?) + }); + let backend: Arc = Arc::new(Mic); + cx.update(|cx| init_in(http, models, backend, cx)); + let vi = cx.update(|cx| VoiceInput::global(cx)); + let inserts: Arc>> = Arc::default(); + let reporter: Reporter = Arc::default(); + { + let inserts = inserts.clone(); + let reporter = reporter.clone(); + cx.update(|cx| { + // What the thread view does: its `Insert` handler, on a refusal, reports it. + cx.subscribe(&vi, move |vi, event: &VoiceEvent, cx| { + if let VoiceEvent::Insert { owner, .. } = event { + inserts.lock().expect("lock").push(*owner); + let report = *reporter.lock().expect("lock"); + if let Some(report) = report { + vi.update(cx, |vi, _| vi.report_insert_refused(report)); + } + } + }) + .detach() + }); + } + cx.run_until_parked(); + Harness { + vi, + capture, + inserts, + reporter, + _tmp: tmp, + } +} + +fn wait_until( + cx: &mut TestAppContext, + what: &str, + mut done: impl FnMut(&mut TestAppContext) -> bool, +) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + cx.run_until_parked(); + if done(cx) { + return; + } + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +fn outcome(line: &str) -> Option<&str> { + line.split(' ') + .find_map(|field| field.strip_prefix("outcome=")) +} + +// ── the wrong collapse ────────────────────────────────────────────────────── + +#[gpui::test] +async fn insert_report_for_another_owner_leaves_this_line_inserted(cx: &mut TestAppContext) { + let h = setup(cx); + let a = cx.new(|_| ()).entity_id(); + let b = cx.new(|_| ()).entity_id(); + *h.reporter.lock().expect("lock") = Some(Owner(b.as_u64())); + + h.record_until_insert(a, cx); + + let lines = h.recording_lines(); + assert_eq!( + lines.len(), + 1, + "R10.1 one line for A's recording: {lines:?}" + ); + assert_eq!( + outcome(&lines[0]), + Some("inserted"), + "B's report does not touch A's line: {lines:?}" + ); +} + +#[gpui::test] +async fn insert_report_with_nothing_pending_logs_nothing(cx: &mut TestAppContext) { + let h = setup(cx); + let a = cx.new(|_| ()).entity_id(); + h.vi.update(cx, |vi, _| vi.report_insert_refused(Owner(a.as_u64()))); + cx.run_until_parked(); + assert!( + h.recording_lines().is_empty(), + "no recording, no line: {:?}", + h.recording_lines() + ); + + h.record_until_insert(a, cx); + let after_recording = h.recording_lines(); + assert_eq!(after_recording.len(), 1, "{after_recording:?}"); + // The line is gone once logged: a late report finds nothing pending. + h.vi.update(cx, |vi, _| vi.report_insert_refused(Owner(a.as_u64()))); + cx.run_until_parked(); + assert_eq!( + h.recording_lines(), + after_recording, + "a report after the line was logged adds nothing" + ); +} + +// ── the wrong split, then the plain case ─────────────────────────────────── + +#[gpui::test] +async fn insert_report_refused_is_the_recordings_one_line(cx: &mut TestAppContext) { + let h = setup(cx); + let a = cx.new(|_| ()).entity_id(); + *h.reporter.lock().expect("lock") = Some(Owner(a.as_u64())); + + h.record_until_insert(a, cx); + + let lines = h.recording_lines(); + assert_eq!( + lines.len(), + 1, + "R10.1 exactly one line, not an inserted one beside a refused one: {lines:?}" + ); + assert_eq!( + outcome(&lines[0]), + Some("refused"), + "R10.1 a refused insert says so: {lines:?}" + ); + for key in [ + "mode=toggle", + "backend=remote", + "model=", + "audio_ms=", + "transcribe_ms=", + ] { + assert!(lines[0].contains(key), "R10.1 {key}: {}", lines[0]); + } +} + +#[gpui::test] +async fn insert_report_no_report_logs_inserted(cx: &mut TestAppContext) { + let h = setup(cx); + let a = cx.new(|_| ()).entity_id(); + h.record_until_insert(a, cx); + let lines = h.recording_lines(); + assert_eq!(lines.len(), 1, "{lines:?}"); + assert_eq!(outcome(&lines[0]), Some("inserted"), "{lines:?}"); +} + +#[gpui::test] +async fn insert_report_lines_carry_no_transcript(cx: &mut TestAppContext) { + let h = setup(cx); + let a = cx.new(|_| ()).entity_id(); + *h.reporter.lock().expect("lock") = Some(Owner(a.as_u64())); + h.record_until_insert(a, cx); + let lines = log_capture::mine(h.capture); + assert!(!lines.is_empty()); + for line in &lines { + assert!(line.starts_with("voice_input event="), "Q14: {line}"); + for leaked in ["Confidential", "dictation", TRANSCRIPT] { + assert!(!line.contains(leaked), "R10.3 {leaked:?} in {line}"); + } + } +} diff --git a/crates/voice_input/tests/remote.rs b/crates/voice_input/tests/remote.rs new file mode 100644 index 0000000..39e3865 --- /dev/null +++ b/crates/voice_input/tests/remote.rs @@ -0,0 +1,253 @@ +// Story 044, task 3.4 — remote transcription against a fake HTTP client (R8.2, R8.5, R8.6, R8.8, +// R10.3). Integration test: `cargo test -p voice_input --test remote`. +// +// Contract (design.md C1.8): `remote::transcribe(http: Arc, url: &Url, +// key: Option, body: Vec, boundary: &str) -> anyhow::Result`, raced against a +// 60 s timer. +// +// ASSUMED TEST SEAM (not in design.md; one function and one constant): the timeout must be testable +// without waiting a real minute, and `transcribe` takes no executor whose clock a test could pause. +// - `remote::REMOTE_TIMEOUT: Duration` (= 60 s) +// - `remote::transcribe_within(http, url, key, body, boundary, timeout: Duration)`, of which +// `transcribe` is `transcribe_within(.., REMOTE_TIMEOUT)`. +// +// Driven with `futures::executor::block_on` (no gpui scheduler); a watchdog thread turns a hang +// into a failure. +// +// Hostile halves: a key sends `Authorization: Bearer ` AND no key sends no Authorization header +// at all (not an empty bearer); an answer just inside the timeout is accepted AND a silent server +// times out; the error carries the status AND at most 200 bytes of the body AND never the key. + +use std::{ + sync::{Arc, Mutex}, + thread, + time::Duration, +}; + +use futures::{channel::oneshot, future::Either}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Method, Response}; +use url::Url; +use voice_input::remote::{REMOTE_TIMEOUT, transcribe, transcribe_within}; + +const BOUNDARY: &str = "zeo-test-boundary"; + +#[derive(Clone, Debug, Default)] +struct Seen { + method: Option, + uri: String, + content_type: Option, + authorization: Option, + body: Vec, +} + +/// Answers after `delay` (on a helper thread, so no executor clock is involved). +fn fake( + status: u16, + answer: &'static str, + delay: Option, +) -> (Arc, Arc>>) { + let seen: Arc>> = Arc::default(); + let log = seen.clone(); + let http = FakeHttpClient::create(move |_req| { + let log = log.clone(); + async move { + log.lock().expect("lock").push(Seen::default()); + if let Some(delay) = delay { + let (tx, rx) = oneshot::channel::<()>(); + thread::spawn(move || { + thread::sleep(delay); + let _ = tx.send(()); + }); + let _ = rx.await; + } + Ok(Response::builder() + .status(status) + .body(AsyncBody::from(answer))?) + } + }); + (http, seen) +} + +/// Records method, URI and headers before the body is read (the request is consumed after). +fn recording(status: u16, answer: &'static str) -> (Arc, Arc>>) { + let seen: Arc>> = Arc::default(); + let log = seen.clone(); + let http = FakeHttpClient::create(move |req| { + let log = log.clone(); + let header = |name: &str| { + req.headers() + .get(name) + .and_then(|v| v.to_str().ok()) + .map(str::to_string) + }; + let mut entry = Seen { + method: Some(req.method().clone()), + uri: req.uri().to_string(), + content_type: header("content-type"), + authorization: header("authorization"), + body: Vec::new(), + }; + async move { + let mut stream = req.into_body(); + futures::AsyncReadExt::read_to_end(&mut stream, &mut entry.body).await?; + log.lock().expect("lock").push(entry); + Ok(Response::builder() + .status(status) + .body(AsyncBody::from(answer))?) + } + }); + (http, seen) +} + +fn silent() -> Arc { + FakeHttpClient::create(|_req| async move { + futures::future::pending::<()>().await; + unreachable!() + }) +} + +fn url() -> Url { + Url::parse("https://stt.example/v1/audio/transcriptions").expect("url") +} + +fn block_on(fut: impl std::future::Future) -> T { + let (tx, rx) = oneshot::channel::<()>(); + thread::spawn(move || { + thread::sleep(Duration::from_secs(20)); + let _ = tx.send(()); + }); + match futures::executor::block_on(futures::future::select(Box::pin(fut), rx)) { + Either::Left((out, _)) => out, + Either::Right(_) => panic!("hung for 20 s"), + } +} + +#[test] +fn remote_posts_multipart_with_bearer_and_returns_text() { + let (http, seen) = recording(200, r#"{"text": "Hello world."}"#); + let body = b"--zeo-test-boundary\r\nWAVBYTES\r\n--zeo-test-boundary--\r\n".to_vec(); + let text = block_on(transcribe( + http, + &url(), + Some("sk-test-123".into()), + body.clone(), + BOUNDARY, + )) + .expect("2xx with text"); + assert_eq!(text, "Hello world.", "R8.2 the `text` of the answer"); + let seen = seen.lock().expect("lock").clone(); + assert_eq!(seen.len(), 1); + assert_eq!(seen[0].method, Some(Method::POST)); + assert_eq!(seen[0].uri, url().to_string()); + assert_eq!( + seen[0].content_type.as_deref(), + Some("multipart/form-data; boundary=zeo-test-boundary") + ); + assert_eq!( + seen[0].authorization.as_deref(), + Some("Bearer sk-test-123"), + "R8.5" + ); + assert_eq!(seen[0].body, body, "the body is sent as built"); +} + +#[test] +fn remote_without_key_sends_no_authorization_header() { + let (http, seen) = recording(200, r#"{"text": "ok"}"#); + block_on(transcribe(http, &url(), None, b"x".to_vec(), BOUNDARY)).expect("ok"); + let seen = seen.lock().expect("lock").clone(); + assert_eq!( + seen[0].authorization, None, + "R8.8 no header at all, not an empty bearer" + ); +} + +#[test] +fn remote_non_success_status_is_an_error_with_status_and_short_body() { + let long_body: &'static str = + Box::leak(format!("{}TAIL-PAST-200-BYTES", "e".repeat(200)).into_boxed_str()); + let (http, _) = recording(500, long_body); + let err = block_on(transcribe( + http, + &url(), + Some("sk-secret-key".into()), + b"x".to_vec(), + BOUNDARY, + )) + .expect_err("R8.6 a 500 is an error"); + let shown = format!("{err:#}"); + assert!(shown.contains("500"), "names the status: {shown}"); + assert!( + shown.contains("eeee"), + "carries the start of the body: {shown}" + ); + assert!( + !shown.contains("TAIL-PAST-200-BYTES"), + "at most 200 bytes of the body: {shown}" + ); + assert!( + !shown.contains("sk-secret-key"), + "R10.3 never the key: {shown}" + ); +} + +#[test] +fn remote_401_is_an_error_not_an_empty_transcript() { + let (http, _) = recording(401, r#"{"error": "bad key"}"#); + let err = block_on(transcribe( + http, + &url(), + Some("k".into()), + b"x".to_vec(), + BOUNDARY, + )) + .expect_err("401"); + assert!(format!("{err:#}").contains("401")); +} + +#[test] +fn remote_success_without_text_field_is_an_error() { + let (http, _) = recording(200, r#"{"transcript": "wrong field"}"#); + assert!(block_on(transcribe(http, &url(), None, b"x".to_vec(), BOUNDARY)).is_err()); + let (http, _) = recording(200, "not json"); + assert!(block_on(transcribe(http, &url(), None, b"x".to_vec(), BOUNDARY)).is_err()); +} + +#[test] +fn remote_timeout_is_sixty_seconds() { + assert_eq!(REMOTE_TIMEOUT, Duration::from_secs(60)); +} + +#[test] +fn remote_silent_server_times_out() { + let err = block_on(transcribe_within( + silent(), + &url(), + None, + b"x".to_vec(), + BOUNDARY, + Duration::from_millis(300), + )) + .expect_err("R8.6 a silent server times out"); + assert!(format!("{err:#}").contains("timed out"), "{err:#}"); +} + +#[test] +fn remote_answer_inside_the_timeout_is_accepted() { + let (http, seen) = fake( + 200, + r#"{"text": "just in time"}"#, + Some(Duration::from_millis(100)), + ); + let text = block_on(transcribe_within( + http, + &url(), + None, + b"x".to_vec(), + BOUNDARY, + Duration::from_secs(5), + )) + .expect("answered before the timeout"); + assert_eq!(text, "just in time"); + assert_eq!(seen.lock().expect("lock").len(), 1); +} diff --git a/crates/voice_input/tests/remote_hardening.rs b/crates/voice_input/tests/remote_hardening.rs new file mode 100644 index 0000000..35ea624 --- /dev/null +++ b/crates/voice_input/tests/remote_hardening.rs @@ -0,0 +1,304 @@ +// Story 044, task 3.4, fix round 1 — hardening found by the tech review (R10.3, Q14). +// Integration test: `cargo test -p voice_input --test remote_hardening`. +// +// Hostile halves: a fragment is refused AND the refusal does not echo it; a transport error that +// names the URL loses its query whether reqwest rewrote it (`?api_key=REDACTED&token=…`) or +// dropped the fragment, on the send path AND on the 2xx body-read path; a key echoed twice leaves +// no prefix of itself in the excerpt; a 2xx answer over 1 MiB is refused AND one under it is +// accepted; control characters in the excerpt become spaces; `new_boundary` never occurs in the +// WAV AND differs across calls. + +use std::{ + io, + pin::Pin, + sync::Arc, + task::{Context, Poll}, + thread, + time::Duration, +}; + +use futures::{AsyncRead, channel::oneshot, future::Either}; +use http_client::{AsyncBody, FakeHttpClient, HttpClient, Response}; +use url::Url; +use voice_input::remote::{multipart, new_boundary, transcribe, validate_url}; + +const BOUNDARY: &str = "zeo-test-boundary"; +const TOKEN: &str = "T0K3N-QUERY-SECRET"; + +fn block_on(fut: impl std::future::Future) -> T { + let (tx, rx) = oneshot::channel::<()>(); + thread::spawn(move || { + thread::sleep(Duration::from_secs(20)); + tx.send(()).ok(); + }); + match futures::executor::block_on(futures::future::select(Box::pin(fut), rx)) { + Either::Left((out, _)) => out, + Either::Right(_) => panic!("hung for 20 s"), + } +} + +fn answering(status: u16, body: Vec) -> Arc { + FakeHttpClient::create(move |_req| { + let body = body.clone(); + async move { + Ok(Response::builder() + .status(status) + .body(AsyncBody::from(body))?) + } + }) +} + +/// A client whose `send` fails the way reqwest does: naming the URL it failed on. +fn failing_send(message: String) -> Arc { + FakeHttpClient::create(move |_req| { + let message = message.clone(); + async move { Err(anyhow::anyhow!(message)) } + }) +} + +/// A body whose first read fails with `message`, as reqwest's `Kind::Body` error does. +struct FailingBody(String); + +impl AsyncRead for FailingBody { + fn poll_read( + self: Pin<&mut Self>, + _cx: &mut Context<'_>, + _buf: &mut [u8], + ) -> Poll> { + Poll::Ready(Err(io::Error::other(self.0.clone()))) + } +} + +fn failing_body(message: String) -> Arc { + FakeHttpClient::create(move |_req| { + let message = message.clone(); + async move { + Ok(Response::builder() + .status(200) + .body(AsyncBody::from_reader(FailingBody(message)))?) + } + }) +} + +fn assert_no_query(shown: &str) { + for leaked in [TOKEN, "api_key", "REDACTED", "?", "#frag"] { + assert!(!shown.contains(leaked), "R10.3 {leaked:?} leaked: {shown}"); + } +} + +#[test] +fn hardening_fragment_is_refused_without_echoing_it() { + for raw in [ + "https://api.example/v1/audio/transcriptions#frag-SECRET", + "http://127.0.0.1:8080/x#frag-SECRET", + ] { + let err = validate_url(raw).expect_err(raw); + let shown = format!("{err:#}"); + assert!( + !shown.contains("frag-SECRET"), + "the refusal does not echo the fragment: {shown}" + ); + } + validate_url("https://api.example/v1/audio/transcriptions").expect("no fragment is fine"); +} + +#[test] +fn hardening_send_error_loses_a_query_rewritten_by_reqwest() { + let url = Url::parse(&format!( + "https://stt.example/v1/audio/transcriptions?api_key=SECRET&token={TOKEN}" + )) + .expect("url"); + let reqwest_text = format!( + "error sending request for url (https://stt.example/v1/audio/transcriptions?api_key=REDACTED&token={TOKEN})" + ); + let err = block_on(transcribe( + failing_send(reqwest_text), + &url, + None, + b"x".to_vec(), + BOUNDARY, + )) + .expect_err("send fails"); + let shown = format!("{err:#}"); + assert!( + shown.contains("https://stt.example/v1/audio/transcriptions"), + "still names the host: {shown}" + ); + assert_no_query(&shown); +} + +#[test] +fn hardening_send_error_loses_the_query_of_a_fragment_less_form() { + let url = Url::parse(&format!( + "https://stt.example/v1/audio/transcriptions?token={TOKEN}#frag" + )) + .expect("url"); + let reqwest_text = format!( + "error sending request for url (https://stt.example/v1/audio/transcriptions?token={TOKEN}): refused" + ); + let err = block_on(transcribe( + failing_send(reqwest_text), + &url, + None, + b"x".to_vec(), + BOUNDARY, + )) + .expect_err("send fails"); + let shown = format!("{err:#}"); + assert!( + shown.contains("refused"), + "the cause after the URL survives: {shown}" + ); + assert_no_query(&shown); +} + +#[test] +fn hardening_body_read_error_loses_the_query() { + let url = Url::parse(&format!( + "https://stt.example/v1/audio/transcriptions?api_key=SECRET&token={TOKEN}" + )) + .expect("url"); + let reqwest_text = format!( + "error decoding response body for url (https://stt.example/v1/audio/transcriptions?api_key=REDACTED&token={TOKEN})" + ); + let err = block_on(transcribe( + failing_body(reqwest_text), + &url, + None, + b"x".to_vec(), + BOUNDARY, + )) + .expect_err("body read fails"); + assert_no_query(&format!("{err:#}")); +} + +#[test] +fn hardening_key_echoed_twice_leaves_no_prefix_of_it() { + let key = "KEYPREFIX-0123456789-abcdefghijklmnopqrstuvwxyz-END"; + // First echo at byte 0, second one straddling where the excerpt is cut. + let mut body = format!("{key} was refused "); + while body.len() < 230 { + body.push('x'); + } + body.push_str(key); + body.push_str(" again"); + let url = Url::parse("https://stt.example/v1/audio/transcriptions").expect("url"); + let err = block_on(transcribe( + answering(401, body.into_bytes()), + &url, + Some(key.into()), + b"x".to_vec(), + BOUNDARY, + )) + .expect_err("401"); + let shown = format!("{err:#}"); + assert!(shown.contains("401"), "{shown}"); + assert!( + shown.contains("[redacted] was refused"), + "the first echo is redacted: {shown}" + ); + assert!( + !shown.contains("KEYP"), + "R10.3 no prefix of the key: {shown}" + ); +} + +#[test] +fn hardening_success_body_over_one_mib_is_refused() { + let url = Url::parse("https://stt.example/v1/audio/transcriptions").expect("url"); + let huge = format!("{{\"text\": \"{}\"}}", "a".repeat(1024 * 1024)); + let err = block_on(transcribe( + answering(200, huge.into_bytes()), + &url, + None, + b"x".to_vec(), + BOUNDARY, + )) + .expect_err("over the cap"); + assert!(format!("{err:#}").contains("larger than 1 MiB"), "{err:#}"); + + let large = format!("{{\"text\": \"{}\"}}", "a".repeat(900 * 1024)); + let text = block_on(transcribe( + answering(200, large.into_bytes()), + &url, + None, + b"x".to_vec(), + BOUNDARY, + )) + .expect("under the cap"); + assert_eq!(text.len(), 900 * 1024); +} + +#[test] +fn hardening_control_characters_in_the_excerpt_become_spaces() { + let url = Url::parse("https://stt.example/v1/audio/transcriptions").expect("url"); + let body = b"\r\nline2\x1b[31m red\tend\x07".to_vec(); + let err = block_on(transcribe( + answering(500, body), + &url, + None, + b"x".to_vec(), + BOUNDARY, + )) + .expect_err("500"); + let shown = format!("{err:#}"); + assert!( + !shown.chars().any(char::is_control), + "Q14 one line: {shown:?}" + ); + assert!(shown.contains(" line2 [31m red end"), "{shown:?}"); +} + +#[test] +fn hardening_new_boundary_is_absent_from_the_wav_and_random() { + let wav = voice_input::audio_prep::wav_pcm16(&[0.0, 0.25, -0.25, 1.0]); + let first = new_boundary(&wav); + let second = new_boundary(&wav); + assert_ne!(first, second, "random across calls"); + assert!( + !first.is_empty() && first.len() <= 70, + "RFC 2046 allows 1..=70 characters: {first}" + ); + + // A WAV that carries the earlier boundaries: the new one is still absent from it. + let mut hostile = wav; + hostile.extend_from_slice(first.as_bytes()); + hostile.extend_from_slice(second.as_bytes()); + for _ in 0..32 { + let boundary = new_boundary(&hostile); + assert!( + !hostile + .windows(boundary.len()) + .any(|window| window == boundary.as_bytes()) + ); + let body = multipart( + &boundary, + &hostile, + "whisper-1", + &voice_input::Language::Auto, + ); + let delimiter = format!("--{boundary}"); + let count = body + .windows(delimiter.len()) + .filter(|window| *window == delimiter.as_bytes()) + .count(); + assert_eq!( + count, 4, + "file, model, response_format and the closing delimiter, nowhere else" + ); + } +} + +// Fix round 2: `scrub` cuts a query at the first `)`, and the `url` crate leaves `)` unencoded, so a +// query holding one would leak its tail; such a URL is refused instead, without echoing the query. +#[test] +fn hardening_query_with_a_closing_parenthesis_is_refused() { + let raw = "http://127.0.0.1:9/v1/audio/transcriptions?token=ab)T0K3N"; + let err = validate_url(raw).expect_err(raw); + let shown = format!("{err:#}"); + assert!(!shown.contains("T0K3N"), "the query is not echoed: {shown}"); + assert!( + validate_url("http://127.0.0.1:9/v1/audio/transcriptions?token=abT0K3N").is_ok(), + "a query without `)` is still accepted" + ); +} diff --git a/crates/zed/Cargo.toml b/crates/zed/Cargo.toml index 6b1750f..28d43be 100644 --- a/crates/zed/Cargo.toml +++ b/crates/zed/Cargo.toml @@ -230,6 +230,7 @@ util.workspace = true uuid.workspace = true vim.workspace = true vim_mode_setting.workspace = true +voice_input.workspace = true watch.workspace = true web_search.workspace = true web_search_providers.workspace = true diff --git a/crates/zed/src/main.rs b/crates/zed/src/main.rs index 1598bc1..80076c4 100644 --- a/crates/zed/src/main.rs +++ b/crates/zed/src/main.rs @@ -510,11 +510,22 @@ fn main() { std::env::consts::ARCH ); let proxy_url = ProxySettings::get_global(cx).proxy_url(); - let http = { + let (http, loopback_http) = { let _guard = Tokio::handle(cx).enter(); - ReqwestClient::proxy_and_user_agent(proxy_url, &user_agent) - .expect("could not start HTTP client") + let http = ReqwestClient::proxy_and_user_agent(proxy_url, &user_agent) + .expect("could not start HTTP client"); + // Voice input sends plain-http requests to a loopback service through this one, so + // they bypass every proxy (R8.9). Without it they are refused behind a proxy. + let loopback_http = ReqwestClient::without_proxy(&user_agent) + .inspect_err(|error| { + log::warn!( + "voice_input event=loopback_client_failed error={:?}", + format!("{error:#}") + ) + }) + .ok(); + (http, loopback_http) }; cx.set_http_client(Arc::new(http)); @@ -712,6 +723,12 @@ fn main() { snippet_provider::init(cx); edit_prediction_registry::init(app_state.client.clone(), app_state.user_store.clone(), cx); let prompt_builder = PromptBuilder::load(app_state.fs.clone(), stdout_is_a_pty(), cx); + voice_input::init(app_state.client.http_client(), cx); + if let Some(loopback_http) = loopback_http { + voice_input::VoiceInput::global(cx).update(cx, |voice_input, _| { + voice_input.set_loopback_http(Arc::new(loopback_http)) + }); + } project::AgentRegistryStore::init_global( cx, app_state.fs.clone(), diff --git a/crates/zed/src/zed.rs b/crates/zed/src/zed.rs index 5d90d59..4c6d8e5 100644 --- a/crates/zed/src/zed.rs +++ b/crates/zed/src/zed.rs @@ -6427,6 +6427,7 @@ mod tests { "toolchain", "variable_list", "vim", + "voice_input", "which_key", "window", "workspace", diff --git a/crates/zlog/src/filter.rs b/crates/zlog/src/filter.rs index afd1466..b7e0397 100644 --- a/crates/zlog/src/filter.rs +++ b/crates/zlog/src/filter.rs @@ -44,6 +44,8 @@ const DEFAULT_FILTERS: &[(&str, log::LevelFilter)] = &[ ("usvg::parser", log::LevelFilter::Error), ("simplecss::selector", log::LevelFilter::Error), ("pet", log::LevelFilter::Warn), + // whisper.cpp reports buffer sizes and model details at info on every model load. + ("whisper_rs", log::LevelFilter::Warn), ]; pub fn init_env_filter(filter: env_config::EnvFilter) {