TESTING: Linux support for Codex Record & Replay demo-to-skill workflows

[joshyorko]

Summary

OpenAI recently added Record & Replay to the Codex app. The feature lets a user demonstrate a workflow on macOS, then Codex turns that demonstration into a reusable Skill.

This issue tracks whether codex-desktop-linux can support the same primitive on Linux, either by directly implementing Linux-native recording or by supporting import/replay of generated skills where possible.

Upstream docs/source-of-truth:

• Record & Replay docs: https://developers.openai.com/codex/record-and-replay
• Skills docs: https://developers.openai.com/codex/skills
• Codex app docs: https://developers.openai.com/codex/app
• Codex changelog: https://developers.openai.com/codex/changelog

Current upstream behavior

Based on the OpenAI Codex docs:

• Record & Replay is currently available on macOS.
• Initial availability excludes the EEA, UK, and Switzerland.
• Computer Use must be available and enabled.
• The feature records a demonstrated workflow and drafts a reusable Codex Skill.
• The generated artifact is conceptually a skill, not just a raw screen recording or coordinate-based macro.

The key point: the capture surface is macOS-specific today, but the output primitive is a Codex Skill. That makes this worth investigating for Linux even if full GUI recording is not immediately possible.

Why this matters for Linux

codex-desktop-linux already exists because the upstream Codex desktop experience is not Linux-native. Record & Replay creates another platform gap:

Capability	macOS	Windows	Linux today
Codex app	supported	supported	not officially native
Computer Use	supported	supported	not officially documented as supported
Record & Replay capture	supported	not currently documented	unsupported
Using reusable skills	should be possible depending on skill content	should be possible depending on skill content	should be possible for CLI/script/browser-oriented skills

This issue should not be scoped as “copy macOS screen recording APIs.” The better framing is:

Can Linux participate in the demo-to-skill workflow, either as a skill consumer, a partial skill replayer, or eventually a native skill authoring environment?

Desired outcome

Design a Linux path for Record & Replay compatibility.

That likely means separating the problem into three layers:

1. Skill compatibility

   • Can codex-desktop-linux discover, load, edit, and invoke Codex Skills generated elsewhere?
   • Are skills just directories with SKILL.md plus optional scripts/assets/references?
   • Where should Linux store/import them?
   • Can we support CLI/script/browser-oriented skills before GUI replay?

2. Replay / execution support

   • Can a generated skill execute through existing Codex CLI, MCP tools, browser automation, or a Linux Computer Use shim?
   • What should happen when a skill requires macOS-only apps or APIs?
   • Can we expose a capability check so unsupported skills fail clearly instead of pretending they work?

3. Linux-native recording / capture

   • What Linux primitives could approximate the macOS recording path?
   • Possible candidates:
     • AT-SPI accessibility APIs
     • X11 event capture
     • Wayland portals / desktop portal APIs
     • compositor-specific protocols
     • browser automation traces
     • app-specific plugin/event streams
   • What can be captured safely and reliably?
   • What permissions model is required?

Non-goals

This should not become a fragile coordinate macro recorder.

Avoid:

• blind mouse-coordinate replay as the main architecture
• pretending Linux GUI automation is equally available across Wayland, X11, GNOME, KDE, Sway, etc.
• hiding permission/security boundaries
• claiming support for arbitrary GUI workflows before the underlying backend exists

The win condition is compatibility with the Codex Skill primitive, not a brittle RPA clone.

Proposed implementation phases

Phase 1: Research and compatibility notes

Document the current upstream Record & Replay assumptions:

• where skills are stored
• how skills are described
• whether generated skills are portable across machines/OSes
• how Codex decides when to invoke a skill
• whether the Linux app can see/use skills created by Codex CLI or another desktop app

Deliverable:

• docs/record-and-replay-linux.md
• clear support matrix
• list of upstream blockers / unknowns

Phase 2: Skill import and invocation

Before attempting Linux recording, support the obvious useful path:

• import or point to an existing Skill directory
• display available skills in the Linux desktop UI
• allow explicit invocation of a skill from a thread
• pass skill context to Codex in the same shape expected by upstream
• fail clearly if a skill requires unsupported GUI/computer-use capabilities

This makes Linux a consumer of Record & Replay output even before Linux can author those skills.

Phase 3: Capability detection

Add a simple capability model for skills:

• cli: can run shell/script instructions
• browser: requires browser automation or page interaction
• desktop-gui: requires Computer Use / screen interaction
• macos-only: requires macOS app paths, AppleScript, Accessibility assumptions, etc.
• windows-only: requires Windows-specific UI/app paths
• linux-gui: future Linux GUI backend

The app should surface this honestly so agents/users know what can run.

Phase 4: Linux capture spike

Only after skill consumption works, spike Linux-native capture.

Research questions:

• Can AT-SPI provide enough semantic UI events for GNOME apps?
• Can browser workflows be recorded more cleanly through browser automation traces instead of global desktop capture?
• Does Wayland require compositor-specific support, portals, or user-approved capture sessions?
• Should this project define a small recorder interface with pluggable backends?

Possible shape:

recording backend
  -> normalized event timeline
  -> observations/screenshots/accessibility tree snapshots
  -> Codex skill-drafting prompt/context
  -> generated Skill directory

Suggested architecture

The clean abstraction is a Linux RecorderBackend rather than one monolithic implementation:

Codex Desktop Linux
  Skills registry
  Skill editor/viewer
  Skill invocation path
  Capability checker
  RecorderBackend trait/interface
    - no-op/import-only backend
    - browser trace backend
    - X11 backend
    - AT-SPI backend
    - future Wayland portal backend

This lets the project ship value early without blocking on the hardest Linux desktop automation problem.

Acceptance criteria

A first useful version of this issue is done when:

• The repo documents current upstream Record & Replay behavior and Linux gaps.
• The app can discover or import a Codex Skill directory.
• The app can show available skills to the user.
• The app can explicitly invoke a skill when it only requires CLI/script/browser-safe capabilities.
• The app detects and clearly explains unsupported GUI/macOS-only skill requirements.
• A follow-up design exists for Linux-native recording backends.

Open questions

• Does upstream expose enough of the generated Skill format to support Linux import cleanly?
• Are Record & Replay skills stored in the same registry as manually-authored Codex Skills?
• Can Codex CLI invoke the same skills without the desktop app?
• How much of Record & Replay skill drafting happens locally versus in upstream Codex services?
• What is the safest Linux permission model for recording desktop workflows?

Bottom line

Record & Replay should be treated as a demo-to-skill compiler, not a screen macro feature.

For Linux, the right first move is probably:

1. support Skill discovery/import/invocation,
2. classify unsupported GUI/platform requirements honestly,
3. then research Linux-native recording backends.

That gives codex-desktop-linux compatibility with the important primitive now, while leaving room for native Linux workflow recording later.

[avifenesh]

@joshyorko You are planning to build it?

[joshyorko]

I guess I could. I'll be real. I don't know if I'm crazy or if it's a problem with the build, but I can't get the Open In file to work for code or code insiders or zed and it's been driving me f****** nuts and so I've been obsessed with it. So I guess I could stop doing that and focus on something more useful like this

[avifenesh]

@joshyorko i saw somebody say ctrl p

[joshyorko]

ok i will start now, i couldnt let it go:

All three are now ready for review:

• [#606](Fix Linux open-target registry helper drift #606)
• [#607](Fix Read Aloud patching for shared app-main bundles #607)
• [#608](Add Linux workspace root Open In targets #608)

[joshyorko]

PR: #611

[joshyorko]

Testing update from the Linux side.

I ran this against a fresh Homebrew-installed Codex Desktop Linux build on GNOME/Wayland, not just against repo tests. The installed app was live from the Homebrew cask path, using upstream app version 26.623.81905 / Electron 42.1.0, with the Record & Replay feature enabled.

What worked in runtime testing:

• The installed Record & Replay backend reported usable capture providers: browser trace, AT-SPI accessibility, screenshot evidence, window metadata, InputCapture/libei readiness, user markers, and speech transcript context.
• Linux Computer Use was usable in the same desktop session: AT-SPI was enabled, the GNOME Shell extension could list/focus windows, screenshot capture worked, and diagnostics reported no blockers.
• A test recording bundle validated cleanly and included the expected evidence shape:
  • transcripts/0000.txt
  • timeline.jsonl with speech_context
  • browser trace artifact with URL/title evidence
  • screenshot artifact
  • accessibility snapshot
  • desktop/window snapshot
  • generated draft-prompt.md
• The generated draft prompt had enough context to describe the demonstrated browser/image/download workflow without relying on coordinate replay or guessing.
• Native audio capture was confirmed opt-in: no recorder starts unless audio is explicitly requested and the CODEX_RECORD_REPLAY_AUDIO gate is affirmative.

I also tested the Skysight/Chronicle-style path separately using a temp runtime directory so it would not pollute my real memory state. That produced:

• segment events.jsonl
• segment metadata.json
• 10-minute activity markdown resources
• a 6-hour rollup markdown resource
• SkysightMemoryInstructions.md
• SkysightSummarizer.md
• browser/window observations
• screenshot and accessibility evidence on an unsuppressed snapshot

Privacy/exclusion behavior also worked in the runtime test. After adding an exclusion for the focused app/window, the next Skysight snapshot emitted suppressed-evidence records and skipped full-screen screenshot/accessibility capture for that excluded visible/focused content.

Current practical takeaway:

Linux can now participate in the demo-to-skill workflow as more than a skill consumer. The useful architecture is semantic evidence capture, not coordinate macro replay: transcript intent, browser trace, window metadata, screenshots, accessibility snapshots, diagnostics, and Chronicle-style memory resources all feed the skill-drafting prompt.

Remaining caveats are mostly environmental rather than conceptual:

• Wayland still needs backend-specific support; on this GNOME session the GNOME Shell extension path is the working window/screenshot route.
• X11 event recording is not generally available on a Wayland session, so it remains metadata/readiness rather than the primary path.
• Arbitrary GUI replay should still be capability-gated. Browser/CLI/script-oriented skills are the cleanest early path; full desktop GUI workflows depend on Computer Use readiness.

Bottom line: the Linux path is viable when treated as a Record & Replay evidence compiler for Codex Skills, with Skysight/Chronicle as the recent-activity substrate.

[joshyorko]

@Yo-DDV @avifenesh would you folks test for me pretty please?

[Yo-DDV]

Hey Josh, no problem, you can count on me.

I have not properly tested all the Record & Replay work yet, but I will take time for it and give you detailed feedback so you have real material to work with. I should be able to get back to you within the next few days, roughly within a week.

[BlockedPath]

@joshyorko I tested it. It ultimately worked on Linux, although I hit a few integration problems first.

Environment

• OS: Ubuntu 26.04 LTS (Resolute Raccoon), x86_64
• Kernel: 7.0.0-22-generic
• Desktop: GNOME Shell 50.1
• Session: Wayland
• Codex Desktop: 26.623.141536
• Electron: 42.1.0
• Record & Replay plugin: 1.0.857
• Event-stream server: 0.1.0-linux-alpha1

Initial problem

Every plugin-managed event-stream call initially failed with:

tool call error: tool call failed for `event-stream/doctor`

Caused by:
    Transport closed

Restarting Codex and re-enabling the plugin did not recover that connection.

The bundled SkyLinuxComputerUseClient binary itself was healthy. Running it directly produced valid diagnostics, and a direct MCP initialization request received a valid response. This suggests the immediate failure was a dead or stale plugin-managed MCP subprocess/connection rather than a broken recorder binary.

Diagnostics inside the Codex sandbox also failed to access the desktop session D-Bus with Operation not permitted. Running the recorder directly outside that sandbox restored access to the desktop portal, AT-SPI bus, and Mutter services.

Workaround

I bypassed the stale plugin MCP connection and started the bundled recorder directly:

SkyLinuxComputerUseClient record start \
  --session-dir <bundle> \
  --goal "Open Chrome and create a GitHub issue" \
  --no-audio

That worked. The recording HUD operated, the session stopped and sealed correctly, and bundle validation passed:

{
  "errors": [],
  "ok": true,
  "warnings": []
}

Codex then generated a reusable github-issue-chrome skill. Skill inspection also passed with no blockers.

Bundle timeline

2026-07-07T21:19:35.104248523Z  session_started
  goal="Open Chrome and create a GitHub issue"

2026-07-07T21:19:35.104622088Z  observation
  backend_catalog recorded

2026-07-07T21:19:35.192679013Z  provider_evidence
  browser-trace available via startup
  file=browser/0000-readiness.json

2026-07-07T21:19:35.193045745Z  provider_evidence
  input-capture-libei available via computer-use-doctor
  file=input-capture/0000-readiness.json

2026-07-07T21:19:35.193372472Z  provider_evidence
  x11-recording missing because this was a Wayland session
  file=x11/0000-session.json

2026-07-07T21:19:35.193698336Z  provider_evidence
  window-metadata missing
  file=x11/0001-window-metadata.json

2026-07-07T21:19:35.228664745Z  diagnostic warning
  GNOME Shell screenshot failed
  Codex GNOME Shell extension screenshot failed
  portal screenshot was denied/cancelled with response code 2
  gnome-screenshot fallback executable was unavailable

2026-07-07T21:19:35.341289122Z  accessibility_snapshot
  file=accessibility/0000.json
  nodes=120

2026-07-07T21:20:49.473364719Z  session_stopped

Diagnostics

Platform:
  linux/x86_64
  Ubuntu GNOME
  XDG_SESSION_TYPE=wayland
  WAYLAND_DISPLAY=wayland-0
  DISPLAY=:0
  GNOME Shell 50.1

Available:
  desktop portal
  RemoteDesktop portal
  ScreenCast portal v5
  Screenshot portal v2
  InputCapture portal
  Mutter RemoteDesktop
  Mutter ScreenCast
  AT-SPI bus
  portal development input

Unavailable or restricted:
  AT-SPI accessibility setting: disabled
  accessibility-tree construction: false
  window querying: false
  focused-window targeting: false
  GNOME Shell GetWindows: AccessDenied
  com.openai.Codex.WindowControl: service unavailable
  /dev/uinput: permission denied
  ydotool/ydotoold: unavailable
  gnome-screenshot executable: unavailable
  X11 recording: unavailable on Wayland

Computed readiness:
  can_register_mcp_tools=true
  can_build_accessibility_tree=false
  can_query_windows=false
  can_focus_apps=true
  can_focus_windows=false
  can_send_development_input=true

Final result

Record & Replay ultimately worked:

• Recorder binary: working
• Direct MCP handshake: working
• Direct CLI recording workaround: working
• Recording HUD and stop control: working
• Bundle creation and sealing: working
• Bundle validation: passed with no errors or warnings
• Skill generation: working
• Skill inspection: passed with no blockers
• Overall demo-to-skill workflow: successful

One limitation remains: the bundle captured startup diagnostics and an accessibility snapshot, but not the detailed Chrome navigation, browser trace, screenshots, or user-action events. The generated skill therefore used the stated goal and a semantic GitHub issue workflow rather than replaying the exact demonstrated browser actions. It was correctly classified as experimental rather than pretending to be a complete coordinate replay.

The main issues appear to be:

1. The plugin-managed MCP connection can become stale and return Transport closed, even though the bundled server remains healthy.
2. A recording started through the direct CLI/HUD path can complete successfully without attaching the ongoing Chrome/browser evidence to the bundle.

So: yes, the feature works and produced a valid reusable skill on this Linux setup, but the plugin reconnection and browser-evidence wiring still need attention.

[Yo-DDV]

@joshyorko thanks for the ping, and thanks @BlockedPath for the detailed Linux run.

I spent time testing this from the Linux side too. My result lines up with part of what BlockedPath saw: the recorder binary itself does not look like the broken piece. Direct CLI runs worked here, including start, mark, speech context, browser trace capture, stop, bundle validation, and draft generation.

One case I could pin down is around stale runtime state. With a stripped-down launch environment, status could fall back to an old status file under /tmp/... instead of the active user runtime dir. After forcing the runtime env to be hydrated before resolving the status path, the same check moved from a stale stopped state to idle under /run/user/1000/..., and the event-stream status reported the same runtime path.

That feels relevant to the stale plugin connection side, but I would still treat it as one failure mode, not the whole story.

Checks I ran on that path: Record & Replay unit tests, Linux feature tests, the window UI patcher tests, workspace cargo check/test, script smoke tests, plus negative cases for symlinked session dirs, duplicate starts, malformed browser traces, stopped sessions, unsupported imports, and expired runtime state.

My read right now: the feature is viable on Linux. The fragile parts are plugin/runtime state recovery and browser evidence attachment. I can keep testing the next patch from this angle if that helps. Thanks again for pushing this, it is a useful feature to get right.