Repository navigation
[Feature]: Play Mode input simulation (manage_input) #1408
Description
Right now an agent can enter Play Mode, read the console and take screenshots, but it can't press a key or click a button. So it can start a game but not actually play it. I'd like to add that.
Proposal: a new manage_input tool (Server/src/services/tools/manage_input.py + MCPForUnity/Editor/Tools/ManageInput.cs), only active in Play Mode:
key: press / hold for N frames / release (Input System keyboard)mouse: move to a screen position, clickui_click: click a uGUI element by GameObject path through the EventSystem
First version would support the Input System package and uGUI only. The old Input Manager can't really be faked from outside, so I'd leave it out and return a clear error when the project uses it.
Tests: EditMode tests for argument checks, plus a small Play Mode test in the test project that presses a key and checks a component saw it.
Two questions before I start:
- Does this fit the project, or would you rather keep tools Editor-only?
- One
manage_inputtool with actions, or split into separate tools?
I already have a docs PR merged here (#1406), so I know the workflow. Happy to adjust the scope.
Activity
akhnafal-aban commentedon Sep 22, 2026
its good
HivemindMinion commentedon Oct 5, 2026
+1 — real-world data point from a mobile project, in case it helps scope this.
Context: Unity 6000.3.8f1, MCP for Unity 10.2.1-beta.6, Android twin-stick shooter (floating move stick on the left thumb, aim/fire stick on the right, tap-to-cast ability buttons). An agent built and wired the whole touch layer this week, but could only verify it by reading code and dumping state. Every actual input path had to be tested by hand on a phone, and three touch bugs only showed up on the device.
Two requests on the proposal:
Touch, including multi-touch. For mobile projects the high-value case is
touchwith several fingers at once: begin finger 0 at A, begin finger 1 at B, move one, lift the other. Most of our bugs lived there (finger-ID ownership when one thumb lifts, a second finger triggering the wrong action, taps on UI that also counted as world input). With the Input System,InputSystem.QueueStateEventon aTouchscreenshould cover it.Legacy Input Manager. Agreed that faking
UnityEngine.Inputfrom outside isn't really possible, so a clear error there makes sense. Butui_clickcould still work for legacy projects: dispatching throughExecuteEvents(pointerDown/Up/Click) against the EventSystem doesn't depend on the input backend, so it would reachStandaloneInputModuleprojects too. A lot of shipping mobile projects are still on the old Input Manager (ours is, deliberately, because "Both" is discouraged on Android), soui_clickworking everywhere would cover menus and HUD buttons even where raw input can't be simulated.On the tool split: one
manage_inputtool with actions sounds right to me, since agents chain these together (hold key, then click) and one schema is easier to plan with.