Task Skill | Agent Skills

Task

Simulate mouse input via Input System in Unity PlayMode: $ARGUMENTS

Workflow

Ensure Unity is in PlayMode (use uloop control-play-mode --action Play if not)
For Click/LongPress: determine the target Game View input position from annotated SimX/SimY, raycast-grid InputX/InputY, or raw image pixels converted with ScreenshotToInputFormula
Execute the appropriate uloop simulate-mouse-input command
Take a screenshot to verify the result: uloop screenshot --capture-mode rendering
Report what happened

Tool Reference

uloop simulate-mouse-input --action <action> [options]

Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | --action | enum | Click | Click, LongPress, MoveDelta, SmoothDelta, Scroll | | --x | number | 0 | Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use AnnotatedElements[].SimX, RaycastGridPoints[].InputX, or raw image pixels converted with ScreenshotToInputFormula. | | --y | number | 0 | Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use AnnotatedElements[].SimY, RaycastGridPoints[].InputY, or raw image pixels converted with ScreenshotToInputFormula. | | --button | enum | Left | Mouse button: Left, Right, Middle. Used by Click and LongPress. | | --duration | number | 0 | Hold duration for LongPress, or interpolation duration for SmoothDelta (seconds). For Click, 0 = one-shot tap. | | --delta-x | number | 0 | Delta X in pixels for MoveDelta/SmoothDelta. Positive = right. | | --delta-y | number | 0 | Delta Y in pixels for MoveDelta/SmoothDelta. Positive = up. | | --scroll-x | number | 0 | Horizontal scroll delta for Scroll action. | | --scroll-y | number | 0 | Vertical scroll delta for Scroll action. Typically 120 per notch. |

Actions

| Action | What it injects | Description | |--------|----------------|-------------| | Click | Mouse.current button press → release | Inject a button click so game logic detects wasPressedThisFrame | | LongPress | Mouse.current button press → hold → release | Hold a button for --duration seconds | | MoveDelta | Mouse.current.delta | Inject mouse movement delta one-shot (e.g. for FPS camera look) | | SmoothDelta | Mouse.current.delta (per-frame) | Inject mouse delta smoothly over --duration seconds (human-like camera pan) | | Scroll | Mouse.current.scroll | Inject scroll wheel input (e.g. for hotbar or zoom) |

Global Options (optional)

| Option | Description | |--------|-------------| | --project-path <path> | Optional. Use only when the target Unity project is not the current directory. |

When to use this vs simulate-mouse-ui

| Scenario | Tool | |----------|------| | Click a Unity UI Button (IPointerClickHandler) | simulate-mouse-ui | | Destroy a block in Minecraft (reads Mouse.current.leftButton) | simulate-mouse-input when the project uses the New Input System | | Place a block with right-click | simulate-mouse-input --button Right when the project uses the New Input System | | Drag a UI slider | simulate-mouse-ui --action Drag | | Look around with mouse (FPS camera) | simulate-mouse-input --action MoveDelta when the project uses the New Input System | | Scroll hotbar slots | simulate-mouse-input --action Scroll when the project uses the New Input System |

Examples

# Left-click at the Game View center (for game logic)
uloop simulate-mouse-input --action Click --x 400 --y 300

# Right-click at screen center (e.g. place block)
uloop simulate-mouse-input --action Click --x 400 --y 300 --button Right

# Hold left-click for 2 seconds (e.g. mine block)
uloop simulate-mouse-input --action LongPress --x 400 --y 300 --duration 2.0

# Look right (FPS camera)
uloop simulate-mouse-input --action MoveDelta --delta-x 100 --delta-y 0

# Scroll up (e.g. previous hotbar slot)
uloop simulate-mouse-input --action Scroll --scroll-y 120

# Scroll down (e.g. next hotbar slot)
uloop simulate-mouse-input --action Scroll --scroll-y -120

# Smooth camera pan right over 0.5 seconds
uloop simulate-mouse-input --action SmoothDelta --delta-x 300 --delta-y 0 --duration 0.5

Coordinate System

--x / --y use top-left Game View coordinates.
Raw image pixels from uloop screenshot --capture-mode rendering must be converted with ScreenshotToInputFormula.
AnnotatedElements[].SimX/SimY and RaycastGridPoints[].InputX/InputY can be passed directly to this tool.
Do not flip Y in the caller. The tool converts internally for Unity Input System:

unity_x = input_x
unity_y = gameViewHeight - input_y

Mouse.current.position uses bottom-left Unity coordinates, so the value read inside Unity may show the converted Y.

Prerequisites

Unity must be in PlayMode
Input System package must be installed (com.unity.inputsystem)
Game code must read input via Input System API (e.g. Mouse.current.leftButton.wasPressedThisFrame)
Use this only when the project already uses the New Input System.

Output

Returns JSON with:

Success: Whether the operation succeeded
Message: Status message
Action: Echoes which action was executed (Click, LongPress, MoveDelta, SmoothDelta, or Scroll)
Button: Which button was used (nullable string; populated for Click / LongPress, null otherwise)
PositionX / PositionY: Target top-left Game View coordinates (nullable float; populated for Click / LongPress)
InputCoordinateSystem: "top-left-game-view" for click/long-press coordinates
UnityCoordinateSystem: "bottom-left-game-view" for the injected Mouse.current.position
GameViewWidth / GameViewHeight: Game View size used for conversion
InputPositionX / InputPositionY: Coordinates received from the caller
InjectedUnityPositionX / InjectedUnityPositionY: Coordinates injected into Mouse.current.position
CoordinateConversionFormula: Conversion formula used by the tool

Verify visual outcome with a follow-up screenshot.

Agent Skills: Task

Install this agent skill to your local

Skill Files