Menu

1 / 5

I Broke a Browser Agent on Purpose. Spotting the Fault Was Not the Same as Fixing It

Reading 0:00
15s threshold

WebFovea , a paper by Jiangang Han, reports a disquieting browser agent failure. A mismatch between screenshot and browser coordinates placed clicks at three quarters of their intended position. A click meant for a point 400 pixels across landed 300 pixels across.

The mismatch came from the service that runs the model, which shrank each screenshot before the model looked at it. The paper estimates that a 1920 by 1080 screenshot reached the model as about 1440 by 810, which is three quarters of the size. Anthropic's vision documentation describes the same behavior for its own models. It says large images are scaled down to cap their cost, and that the model answers in positions on the resized picture.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.