[API] Upgrade UMG MCP to Gemini CLI Skills

[winyunq]

Context

We are planning to upgrade the current UMG MCP (Model Context Protocol) implementation to use the Gemini CLI Skills architecture. This represents a shift from a generic protocol adapter to a native integration with the Gemini CLI.

MCP vs. Skills: A Technical Comparison

Current Approach: MCP (Model Context Protocol)

• Architecture: Client-Server. The Python script runs a standalone server (HTTP/Stdio) that listens for JSON-RPC messages.
• Workflow: User -> Gemini CLI -> MCP Client -> Network/Pipe -> UMG MCP Server (Python) -> UE5.
• Pros: Standardized, client-agnostic.
• Cons: Requires managing a server process, serialization overhead, and connection handling.

Target Approach: Gemini CLI Skill

• Architecture: Native Plugin/Module. The Python logic is loaded or registered directly by the Gemini CLI runtime.
• Workflow: User -> Gemini CLI (Native Function Call) -> Direct Python Call -> UE5.
• Pros: Tighter integration, state persistence is easier to manage within the session, simplified architecture (no separate server loop needed).
• Cons: Specific to Gemini CLI environment.

Proposed Changes

We aim to keep the modification strict to the Python layer, preserving the C++ plugin as-is.

1. Python Layer (Major Changes)

We need to refactor \Resources/Python/.

• Decouple Logic: Ensure existing logic in \UmgMcpServer.py\ and subsystems (Widget, Animation, etc.) is separated from the MCP server boilerplate.
• Skill Wrapper: Create a new entry point (e.g., \UmgMcpSkill.py) that exposes the core functionalities as importable, annotated functions that Gemini CLI can recognize as tools/skills.
• Configuration: Update \mcp_config.py\ or equivalent to support Skill registration parameters.

2. C++ Layer (No Changes Expected)

• The C++ plugin (\Source/UmgMcp/) acts as the command receiver/executor inside Unreal Engine.
• As long as the Python script communicates with UE5 (via the existing socket/bridge mechanism) using the same payload format, no C++ changes are required.

Verification Plan

1. Create a prototype Skill definition.
2. Verify communication with the running UE5 Editor.
3. Compare latency and usability against the old MCP implementation.

[winyunq]

Technical Plan Refinement: "Add-on" Strategy

Based on further analysis and project requirements, we will adopt a non-destructive, add-on strategy. We are not rewriting the core logic but adding a new entry point for Gemini CLI Skills.

1. New Entry Point: UmgMcpSkills.py

Instead of modifying UmgMcpServer.py (which runs a standalone MCP server), we will create a new file Resources/Python/UmgMcpSkills.py.

Role:

• Acts as a Bridge between Gemini CLI's native runtime and the Unreal Engine Socket.
• Reuses the existing UnrealConnection class and subsystems (Widget, Animation, etc.) located in Resources/Python/.
• No dependencies on fastmcp: This file will define pure Python functions with standard type hints and docstrings.

2. Skill Protocol & Definition

We assume the Gemini CLI loads Skills by importing a Python module and inspecting its functions.

Structure of UmgMcpSkills.py:

# Imports from existing logic
from Bridge.UnrealConnection import get_unreal_connection
from Widget import UMGSet

# Skill Definition (Example)
def create_widget(parent_name: str, widget_type: str, new_widget_name: str) -> dict:
    """
    Creates a new widget in the Unreal Editor.
    
    Args:
        parent_name: Name of the parent widget.
        widget_type: Class name (e.g., 'Button').
        new_widget_name: Name for the new widget.
    """
    conn = get_unreal_connection()
    client = UMGSet.UMGSet(conn)
    # Async handling might be needed depending on CLI's execution model (sync vs async)
    return client.create_widget_sync(widget_type, new_widget_name, parent_name) 

3. Configuration & PromptManager

The existing Resources/Python/PromptManager/server.py and prompts.json will remain vital.

• Role of PromptManager: Continues to serve as a GUI for users to enable/disable specific tools and edit prompts.
• Integration: UmgMcpSkills.py will load prompts.json on initialization.
  • If a tool is disabled in JSON, the corresponding function in UmgMcpSkills.py can either be hidden (if dynamic dispatch is supported) or simply return a "Feature Disabled" message.

4. Next Steps for Implementation

1. Draft UmgMcpSkills.py: Implement a few core skills (e.g., get_widget_tree, create_widget) to test the connection.
2. Async/Sync Adaptation: The current UmgMcpServer.py uses asyncio. We need to confirm if Gemini CLI Skills support async functions. If not, we'll need a synchronous wrapper for the UnrealConnection.

[winyunq]

Guide: Configuring and Customizing UMG Skills

This section outlines how users and developers will interact with the new Skill system, focusing on configuration via PromptManager and code customization.

1. The Skill Entry Point

• File: Resources/Python/UmgMcpSkills.py (Proposed New File)
• Purpose: This acts as the single source of truth for Gemini CLI. It exposes Python functions (e.g., create_widget, get_widget_tree) that the CLI recognizes as "Skills".

2. Managing Skills via PromptManager

We will leverage the existing prompts.json ecosystem to provide a configuration layer on top of the raw Python code.

• Config File: Resources/prompts.json
• GUI Tool: Run Resources/Python/PromptManager/server.py to open the web interface.
• Mechanism:
  1. Toggle Features: Users can uncheck tools in the PromptManager UI. This updates prompts.json.
  2. Dynamic Binding: When UmgMcpSkills.py loads, it reads prompts.json.
     • If a tool is disabled in JSON, the corresponding Python function in UmgMcpSkills.py will be skipped (not registered) or replaced with a dummy function, effectively hiding it from the AI.
  3. Prompt/Docstring Injection: The Description field in prompts.json will be dynamically injected as the docstring of the Python function. This allows users to tune the AI's understanding of a skill without touching the code.

3. How to "Edit" (Customize) Skills

Scenario A: I want to change how a skill works.

• Action: Edit Resources/Python/UmgMcpSkills.py.
• Detail: Find the function (e.g., create_widget) and modify its implementation (e.g., change default arguments or add logging).

Scenario B: I want to add a completely new skill.

• Action:
  1. Open Resources/Python/UmgMcpSkills.py.
  2. Define a new function with Type Hints.

  def my_custom_tool(param: str) -> str:
      """Description for the AI."""
      # Logic here

  3. (Optional) Add it to prompts.json via the PromptManager if you want to enable/disable it dynamically.

4. Integration Summary

Component	Responsibility
UmgMcpSkills.py	The Body. Contains the actual logic and functions.
prompts.json	The Brain Config. Controls which skills are active and how they are described to the AI.
PromptManager	The Interface. A user-friendly way to edit the Brain Config.

[winyunq]

Client-Side Configuration (Gemini CLI)

To enable the new Skill, the user must update their local Gemini CLI configuration file (typically %USERPROFILE%\.gemini\settings.json on Windows).

Configuration Schema

Unlike mcpServers, which run as separate processes, skills are loaded directly.

Add the following to settings.json:

{
  "skills": {
    "UnrealMotionGraphicsMCP": {
      "path": "<YOUR_PROJECT_PATH>/Plugins/PublicUmgMcp/Resources/Python/UmgMcpSkills.py",
      "description": "Direct control of Unreal Engine UMG via Python Skills",
      "type": "local"
    }
  }
}

Note: Replace <YOUR_PROJECT_PATH> with the actual absolute path to your Unreal Engine project root.
Example (Windows): "D:/Projects/MyGame/Plugins/PublicUmgMcp/Resources/Python/UmgMcpSkills.py"
Tip: Use forward slashes / or double backslashes \\ for Windows paths.

Key Differences from MCP Config

1. Key Name: Uses "skills" instead of "mcpServers".
2. Path: Points directly to the .py file, not a command to run a server.
3. Performance: Since it's loaded locally, there is zero network overhead compared to the local host networking of MCP.

[winyunq]

Progress update, not closing yet.

The repository now has an umg-mcp-authoring Skill that records target/default usage, HLSL material workflow, Sequencer workflow, Blueprint transitional guidance, and the five-system login demo validation path. Recent commits also added protocol documents and static checks for Material, Animation, and Blueprint surfaces.

This issue originally asks for a deeper Gemini CLI Skills architecture shift in the Python layer. That is not complete: the runtime path is still MCP stdio -> Python -> UE socket, with Skill used as experience/instructions rather than a direct Python-call replacement. Keeping this open is appropriate until we decide whether to actually refactor the Python server layer.

[winyunq]

Progress update.

The repo now has an umg-mcp-authoring Codex Skill plus runnable demo/test cases, including the five-system login demo script and per-system protocol checks. The latest UMG widget update also teaches the Skill to use move_widget and avoid hidden compatibility paths such as reparent_widget.

Keeping this issue open because the original discussion also considered a deeper Gemini CLI/direct Python skill entry point (UmgMcpSkills.py). Current implementation treats Skill as experience/instructions over the MCP protocol, not as a replacement runtime API.

[winyunq]

Progress update, not closing.

The Skill side was updated alongside the bluecode MVP:

• Skills/umg-mcp-authoring now documents the bluecode workflow instead of the legacy node-by-node Blueprint flow.
• UmgMcpSkills.py now includes direct wrappers for the bluecode functions so Skill-style direct calls do not fall through to unknown bluecode_* socket commands.
• load_skills() now prefers concrete Python callables when present, then falls back to socket wrappers.
• The five-system login demo guidance and script now use bluecode_set_function, bluecode_apply, bluecode_read_function, and bluecode_compile.

This still does not close the issue. The original issue proposed a deeper Gemini/direct Python Skill architecture that could replace the MCP server runtime. Current state is still MCP/stdio + UE socket as the real transport, with Skill used as experience/instructions plus a partial direct wrapper layer.

[winyunq]

Progress update, not closing.

Pushed another Skill/documentation refinement on codex/login-demo-skill:

• Public: 8dfa582 (Document Sequencer MCP runtime validation)
• Fab base: 3d9d8a9

This records the Sequencer runtime validation boundary in README/Readme_zh, Document/AnimationMcpProtocol.md, and Skills/umg-mcp-authoring:

• Sequencer validation must run through stdio MCP (Sequencer_RenderTransform_Check.py), not a raw socket bypass.
• The runtime check covers default tool exposure, widget-track writes, time-slice writes, overview/widget/time semantic reads, UMG 2D transform tracks, and confirmed deletion behavior.
• UE-backed APITest scripts should run sequentially because one editor process shares Active Target/scope state.

Runtime evidence from this pass: Sequencer_RenderTransform_Check.py passed against UE5.8 via MCP stdio, with 42 MCP tools, overview_track_count=6, semantic time-slice values for RenderOpacity, RenderTransform.Translation, RenderTransform.Angle, and RenderTransform.Scale.X/Y, plus delete confirmation enforcement.

Keeping this issue open because the direct Skill/runtime architecture question is still broader than these Skill instructions and validation paths.