path: root/SPEC.md

# Parley specification

## 1. Purpose

Parley is a local-first dictation tool for GNOME on Wayland. It turns microphone recordings into text with the existing Parakeet transcription runtime and inserts the result at the focused application's caret when the desktop permits it.

The normal interaction is:

1. Place the caret in an editable field.
2. Press a global shortcut or click the top-bar microphone.
3. Speak.
4. Press the shortcut or click again.
5. Parley transcribes locally.
6. Parley inserts through the configured insertion backend.
7. Parley also saves the transcript and copies it to the clipboard.

The supported target environment is deliberately narrow:

- fully updated Arch Linux
- GNOME Shell 50 on Wayland
- IBus 1.5.34
- PipeWire/WirePlumber with PulseAudio compatibility
- Python 3.14 with PyGObject
- The existing `transcribe-cli` binary and Parakeet Unified EN model

## 2. Goals

- Feel native on GNOME: top-bar status, global shortcut, notifications, and clear recording/transcribing states.
- Work in as many desktop applications as Wayland and application input-method support allow.
- Keep recording and transcription entirely local.
- Preserve every successful transcript even when insertion fails.
- Keep GNOME Shell responsive and isolated from model, audio, and subprocess failures.
- Provide useful operation before the Shell extension is complete.
- Keep insertion backends replaceable behind one interface.
- Continue to offer a standalone CLI frontend to the shared transcription core.

## 3. Non-goals and platform limits

- Parley cannot guarantee insertion into literally every application. Lock/login screens, games, custom-rendered controls, unsupported XWayland clients, and applications without input-method support may reject insertion.
- Parley will not bypass the lock screen or cross user/session boundaries.
- Continuous live captioning and partial streaming text are not required for v1.
- Flatpak packaging is not a v1 requirement. A Flatpak would require a separate portal-oriented audio and filesystem design.
- Cloud transcription is out of scope.
- The Shell extension will not load the model, capture audio, or run transcription.

## 4. Transcription pipeline

The terminal-independent Python core now:

- records the default input with FFmpeg's `pulse` input at mono 16 kHz PCM;
- invokes `transcribe-cli` as a cancellable child process;
- uses `parakeet-unified-en-0.6b-Q8_0.gguf` (approximately 698 MiB);
- parses JSONL batch output;
- persists UTF-8 transcripts and their WAV recordings together; and
- exposes the same core to the standalone terminal frontend and daemon.

Development can use runtime assets from the sibling `transcribe-parakeet` checkout. Installed assets default to `~/.local/bin/transcribe-cli` and `~/.local/share/parley/models/`. The current runtime binary links to CUDA 13 and Intel MKL libraries. Packaging must declare those host dependencies or later supply a more portable build.

## 5. Architecture

```text
GNOME Shell extension          toggle CLI / custom shortcut
          |                              |
          +------------- D-Bus ----------+
                         |
                         v
              Parley per-user daemon
              - state machine
              - audio recording
              - transcription
              - transcript history
              - clipboard/notifications
              - insertion policy
                         |
          +--------------+----------------+
          |              |                |
          v              v                v
        IBus       RemoteDesktop/       Clipboard
                    libei paste
```

### 5.1 Shared core

Terminal-independent Python modules own:

- recording lifecycle;
- transcription invocation and JSON parsing;
- transcript persistence;
- configuration;
- errors and cancellation.

The existing CLI behavior becomes one frontend to this core. No reusable core function may read raw terminal keys or require a TTY.

### 5.2 Per-user daemon

A long-running Python/GObject process owns mutable state and serializes operations. It runs as a systemd user service and is activated on demand where practical.

Only one recording/transcription operation may run at a time. Long-running work and subprocess waits must not block the GLib main loop.

Provisional D-Bus identity:

- bus name: `org.parley.Transcription1`
- object path: `/org/parley/Transcription1`
- interface: `org.parley.Transcription1`

The name can be finalized before public packaging.

Minimum methods:

- `StartRecording()`
- `StopAndTranscribe()`
- `Toggle()`
- `Cancel()`
- `CopyLastTranscript()`
- `InsertLastTranscript()`
- `OpenTranscriptFolder()`

Minimum read-only properties:

- `State`: `idle`, `recording`, `transcribing`, `inserting`, or `error`
- `LastTranscript`
- `LastTranscriptPath`
- `LastError`
- `InsertionMode`
- `AutoInsert`

Minimum signals:

- `StateChanged(state)`
- `TranscriptReady(text, path)`
- `Error(code, message)`
- `InsertionFinished(backend, success, message)`

Exact D-Bus signatures and error names will be versioned in an interface XML file during implementation.

### 5.3 State machine

Normal flow:

```text
idle -> recording -> transcribing -> inserting -> idle
```

If automatic insertion is disabled or unavailable:

```text
idle -> recording -> transcribing -> idle
```

Rules:

- `Toggle` starts from `idle` and stops from `recording`.
- A toggle while transcribing or inserting must not start a second operation.
- `Cancel` during recording discards temporary audio.
- Cancellation during transcription should terminate the child process cleanly, escalating only when necessary.
- Errors transition through `error`, emit details, and leave the daemon recoverable without a restart.
- Temporary files are removed after completion or cancellation.
- A successful transcript is persisted before insertion is attempted.

### 5.4 systemd user service

The daemon is managed by a user unit. The unit must:

- restart only on genuine crashes, with rate limiting;
- inherit the graphical session environment needed for D-Bus, PipeWire, IBus, and portals;
- stop child recording/transcription processes on shutdown; and
- log structured, actionable messages to the user journal without logging microphone audio.

### 5.5 Thin clients

A small command-line client calls the D-Bus service. It supports at least:

```text
parleyctl toggle
parleyctl start
parleyctl stop
parleyctl cancel
parleyctl status
parleyctl copy
parleyctl insert
```

This client is used by the early GNOME custom keyboard shortcut and remains a fallback if the Shell extension is disabled by an upgrade.

## 6. Recording and transcription

### 6.1 Recording v1

Keep the proven FFmpeg path through `pipewire-pulse`:

- input: `pulse`, device `default`;
- mono;
- 16 kHz;
- signed 16-bit PCM WAV;
- configurable maximum duration;
- controlled shutdown that produces a valid WAV;
- clear handling for missing/default-device changes.

Direct PipeWire capture may be evaluated later but is not required merely for architectural purity.

### 6.2 Transcription v1

Invoke the existing binary with the current model and options unless testing justifies a change:

- quiet output;
- language `en`;
- no timestamps;
- 30 ms stream chunks;
- JSONL batch output.

Binary and model paths are configuration, not hard-coded repository assumptions. Development may point at the sibling prototype's assets. Distribution should not duplicate the model unnecessarily.

### 6.3 Persistence

- Save every successful transcript and its WAV recording before insertion.
- Group recordings by local date and save each pair as `HH-MM-SS.txt` and `HH-MM-SS.wav`.
- Store user data under an XDG-appropriate data directory by default; support a configurable transcript directory.
- Write UTF-8 text with a final newline.
- Keep `LastTranscript` and its path available over D-Bus.
- A history UI is optional for v1, but the storage layout must not prevent one.

### 6.4 Performance

The initial implementation may launch `transcribe-cli` once per utterance. Reloading the approximately 700 MiB model is expected to be the largest latency cost.

After correctness is established, evaluate a persistent worker or a library/server API that keeps the model loaded. The daemon and D-Bus contracts must not depend on one-shot subprocesses so this can change internally.

## 7. Insertion strategy

All insertion implementations conform to one daemon-side abstraction, conceptually:

```python
insert_text(text, context) -> InsertionResult
```

An `InsertionResult` identifies the backend, success/failure, and a user-safe explanation. Backend selection is configurable and must degrade predictably.

Regardless of selected backend, a successful transcript is saved and copied to the clipboard before automatic insertion is attempted.

### 7.1 Primary candidate: IBus `commit_text`

A minimal persistent passthrough IBus engine is implemented. When it owns the focused input context, it commits the transcript as semantic text. While transcription is running, it uses temporary IBus preedit text for an animated in-field processing indicator; the indicator is never inserted into the document. Initial end-to-end insertion and ordinary passthrough typing work across the applications recorded in the bake-off matrix; the full input-behavior matrix is still required.

Benefits:

- follows the desktop input-method path rather than simulating hardware;
- correct Unicode and multiline handling;
- no keyboard-layout translation;
- fewer focus races than clipboard plus a synthetic paste shortcut;
- no remote-control permission prompt.

The engine must be active for the focused input context. Parley therefore uses temporary activation for each dictation session: it remembers the current engine, selects Parley while recording and transcribing, commits semantic text, and restores the previous engine after insertion, cancellation, or failure. If GNOME changes input sources during the session, Parley records the latest choice, reacquires its engine, and restores that choice afterward. The committed transcript is Unicode text and does not depend on the active keyboard layout.

### 7.2 Safe fallback: clipboard only

Copy the transcript and notify the user to press Ctrl+V. This requires one user action but does not synthesize input and should always remain available.

### 7.3 Compatibility fallback: RemoteDesktop portal plus libei

Use the XDG Desktop Portal RemoteDesktop API to request keyboard control and libei/EIS to synthesize a paste shortcut after placing the transcript on the clipboard.

Requirements:

- request only the capability needed;
- support GNOME restore tokens and persist them securely when returned;
- handle missing, expired, denied, or revoked permission without losing text;
- clearly expose when a remote-control session is active;
- do not assume persistence is guaranteed on every portal backend or forever;
- recreate the session when required; and
- fall back to clipboard-only on any failure.

This avoids root/uinput configuration and is preferred over `ydotool` for synthetic paste. It still has focus races, depends on the target accepting Ctrl+V, and cannot insert arbitrary Unicode directly—it pastes clipboard contents.

### 7.4 Last resort: `ydotool`

An opt-in debugging/compatibility backend may synthesize Ctrl+V through `ydotool`. It is never the default because it requires access to `uinput`, may involve a privileged helper or device rules, weakens the Wayland security boundary, and retains synthetic-input focus races.

### 7.5 Backend policy

The insertion bake-off determines the shipped default. Intended preference order is:

1. IBus, if reliable;
2. clipboard only as the universally safe fallback;
3. RemoteDesktop/libei as an explicitly enabled compatibility mode;
4. `ydotool` as an explicitly enabled last resort.

The UI may offer `ibus`, `clipboard`, `portal-paste`, and `ydotool-paste` modes. An `auto` mode may use IBus and degrade to clipboard, but it must not unexpectedly open a remote-control permission dialog or start synthetic input without prior user choice.

## 8. GNOME integration

### 8.1 Early global shortcut

Before the extension exists, install/document a GNOME custom keyboard shortcut that runs `parleyctl toggle`. This provides an end-to-end workflow early:

```text
hotkey -> record -> hotkey -> transcribe -> save/copy -> manual paste
```

Installation must not overwrite an existing conflicting shortcut without confirmation. The shortcut command remains supported after the extension ships.

### 8.2 GNOME Shell extension

A GNOME Shell 50 extension provides:

- a top-bar microphone indicator;
- idle, recording, transcribing, inserting, and error presentation;
- click to start/stop;
- cancel action;
- copy and insert-last actions;
- open-transcript-folder action;
- automatic insertion toggle;
- insertion mode display/selection or a settings link;
- notifications; and
- an extension-managed global shortcut.

The extension is a D-Bus client only. It must use documented GNOME Shell extension APIs wherever available, declare compatible Shell versions explicitly, and tolerate daemon unavailability.

The extension shortcut replaces or disables the development custom shortcut to prevent duplicate activation, while `parleyctl` remains usable as a recovery path.

## 9. Notifications and clipboard

- Copy every successful transcript to the clipboard.
- Desktop notifications are disabled by default and can be explicitly enabled.
- When enabled, notify on transcription completion when no visible UI feedback is available.
- When enabled, notify when insertion falls back to manual paste.
- Surface concise errors while retaining details in the journal.
- Avoid repeated noisy notifications when the top-bar state already makes progress obvious.
- Use a registered desktop application identity for production notifications; `notify-send` is acceptable during the prototype phase.

## 10. Configuration

Daemon configuration uses the optional TOML file `~/.config/parley/config.toml` with system-oriented defaults. Parley does not use application-specific environment variables. GNOME Shell-only preferences may use GSettings when schemas are introduced.

Planned settings:

- microphone/device;
- model and runtime paths;
- language;
- transcript directory;
- maximum recording duration;
- automatic insertion enabled;
- insertion backend;
- notification and sound preferences;
- shortcut;
- optional punctuation/formatting behavior.

Secrets and portal restore tokens must not be exposed over D-Bus or written to logs. Restore-token storage should follow portal guidance; if it is not itself sensitive, it must still be treated as user-private state.

## 11. Security and privacy

- Audio and text remain local unless the user explicitly copies or pastes them into another service/application.
- Audio from successful transcriptions is retained beside its transcript. Temporary audio is deleted after cancellation or error.
- The recording state must always be visibly indicated.
- Do not log transcript contents by default.
- Portal control is opt-in and revocable.
- `ydotool` support is opt-in and documents its elevated input capability.
- D-Bus methods are available only on the user's session bus; no system-wide service is needed.

## 12. Delivery phases

### Phase 1 — Refactor (complete)

- Create the terminal-independent core.
- Implement start/stop/cancel recording operations.
- Extract transcription and persistence.
- Preserve the current CLI as a frontend.
- Add unit tests around state-independent parsing, paths, and errors.

Exit criterion: CLI behavior works through the shared core without core TTY dependencies.

### Phase 2 — Daemon, D-Bus, and early hotkey (complete for development)

- Implement the GLib daemon and state machine.
- Add the systemd user unit.
- Add `parleyctl`.
- Save and copy successful transcripts.
- Add prototype notifications.
- Configure a GNOME custom shortcut for `parleyctl toggle`.

Exit criterion: hotkey-driven recording/transcription reliably ends with a saved transcript on the clipboard.

### Phase 3 — Insertion bake-off (in progress)

Build two deliberately small prototypes:

- IBus `commit_text`, testing persistent passthrough and temporary activation;
- RemoteDesktop portal/libei paste, including permission denial and restore-token reuse.

Test both against:

- GNOME Text Editor or another GTK 4 editor;
- Firefox;
- Chromium;
- LibreOffice;
- GNOME Console/Terminal;
- VS Code or another Electron application;
- a Qt application;
- representative XWayland applications;
- multiple keyboard layouts if available.

For IBus passthrough, explicitly test ordinary typing, modifiers, application shortcuts, compose/dead keys, layout switching, and interaction with any real IM engines. For temporary activation, test caret/focus retention and engine restoration.

For portal paste, test first authorization, restored sessions, logout/login, permission revocation, focus changes, multiline/Unicode clipboard content, and applications that remap or reject Ctrl+V.

Exit criterion: record results, choose the default backend, and document unsupported applications. If IBus passthrough or activation is flaky, use clipboard as the safe default and retain portal paste as opt-in.

### Phase 4 — GNOME Shell extension (baseline implemented)

- Implement the top-bar indicator and menu.
- Synchronize state over D-Bus.
- Add notifications and the extension shortcut.
- Keep `parleyctl` and the custom-shortcut setup available as recovery tools.

Exit criterion: all daemon states and failures are represented without blocking or destabilizing GNOME Shell.

### Phase 5 — Production insertion backends

- Turn successful prototypes into selectable backends.
- Implement deterministic fallback behavior.
- Ensure every failed insertion leaves text saved and copied.

### Phase 6 — UX, performance, and packaging

- Evaluate a persistent loaded model.
- Add microphone selection and maximum duration UI.
- Add optional push-to-talk behavior where reliable key-release handling is available.
- Add punctuation/formatting commands.
- Package the daemon, D-Bus interface, systemd unit, desktop metadata, GSettings schema, IBus component, CLI, and extension.
- Document CUDA/MKL requirements or provide a portable runtime build.

## 13. Testing requirements

### Automated

- state-machine transitions and invalid calls;
- cancellation and child-process cleanup;
- JSONL parsing, including malformed/no-result/error rows;
- dated transcript and WAV pair persistence;
- D-Bus methods, properties, signals, and typed errors;
- insertion fallback selection;
- configuration migration/defaults;
- no transcript loss on insertion failure.

Runtime and desktop dependencies should be abstracted so core tests do not require a microphone, GNOME Shell, IBus, or the 698 MiB model.

### Manual integration

- repeated start/stop/cancel cycles;
- microphone removal and unavailable input;
- daemon restart and Shell extension restart;
- GNOME logout/login;
- GPU/runtime failures;
- long recordings and maximum-duration cutoff;
- the Phase 3 application matrix;
- focus changes while transcribing;
- clipboard ownership after daemon exit/restart;
- extension disabled after a GNOME upgrade.

## 14. Acceptance criteria for the first useful release

- A keyboard shortcut starts and stops recording.
- Recording and transcription do not block GNOME Shell.
- Every successful transcript is saved and copied.
- The selected insertion backend inserts in the supported application matrix or reports a clear fallback.
- Cancellation and errors leave the service ready for another recording.
- A top-bar indicator accurately shows recording and processing states.
- The tool works without network access.
- Disabling or breaking the Shell extension does not remove CLI/hotkey access to the daemon.

## 15. Open decisions

These must be resolved by prototypes or packaging work rather than assumption:

- whether the promising IBus persistent passthrough results remain reliable across the full test matrix;
- exact compatibility of IBus and portal paste across the target application's real versions;
- GNOME portal restore-token lifetime and user experience across login sessions;
- final reverse-DNS application/D-Bus identifier;
- persistent model API/process design;
- portable versus host-specific CUDA/MKL distribution;
- direct PipeWire capture versus keeping FFmpeg/PulseAudio compatibility;
- whether push-to-talk can use stable public GNOME APIs on Shell 50.