Building an Android AI Agent, Part 10: Handling Unexpected Screens Mid-Task

In Part 9 we built a flat task executor that walks a step list with a cursor, pauses for confirmation, and keeps state on the client.

The assumption baked into that executor: when the agent arrives at a step, the expected screen is there.

It often isn't.

A permission dialog appears. The app updated overnight and the UI changed. A loading spinner is still running. A previous step succeeded but triggered a side effect — an ad, a rate prompt, a "what's new" overlay.

The agent needs to handle this without asking the user every time, and without silently doing the wrong thing.

What "unexpected" actually means

There are three distinct cases:

Transient obstruction — a dialog, overlay, or spinner that wasn't part of the plan but can be dismissed or waited out.…