Menu

1 / 2

# Giving an LLM Eyes and Hands on a Mobile Simulator

DEV Community: ios·Duchan·5 months ago
#91bVwkIz
#dev#model#screenshot#server#article
Reading 0:00
15s threshold

The interface a human uses

When a person does QA in tapflow, the loop is:

Look at the simulator screen

Decide what to do (tap, swipe, type)

Do it

Look again

This is exactly the perception-action loop that vision-capable LLMs are built for. The model sees a screenshot, reasons about what it shows, decides what action to take, and calls a tool to execute it.

We didn't need to build a new automation layer. We just needed to expose tapflow's existing WebSocket and REST APIs as MCP tools.

What the MCP server does

@tapflowio/mcp-server connects to a running tapflow relay and registers 13 tools that any MCP-compatible client can call:

list_devices — see all simulators registered on the relay connect_device — join a device session boot_device — boot a simulator (waits up to 30s for ready state) screenshot — capture the current screen tap — tap at a pixel coordinate swipe — swipe between two coordinates type_text — type into the focused field press_key — press a keyboard key (Return, Delete, Escape...)…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.