Plover: Steering GUI Agents through Plan-Centric Interaction
Plover is a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. It uses a planner-executor architecture to support explicit supervision, localized correction, natural-language guidance, and screenshot-grounded interventions. A formative study with six participants informed the interaction design. Evaluation through benchmark failure-case repair and scenario-based workflow analyses shows that many autonomous GUI-agent failures are structurally repairable when plans remain visible and intervent…
Development
- First ReportPlover: Steering GUI Agents through Plan-Centric InteractionarXiv cs.AI
- Current AssessmentThe approach addresses a key limitation of current GUI agents—lack of user oversight—by making plans inspectable and revisable, which could influence future product design in automation tools.Agent Pulse · analysis
Plover is a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. It uses a planner-executor architecture to support explicit supervision, localized correction, natural-language guidance, and screenshot-grounded interventions. A formative study with six participants informed the interaction design. Evaluation through benchmark failure-case repair and scenario-based workflow analyses shows that many autonomous GUI-agent failures are structurally repairable when plans remain visible and intervent…
Plover's planner-executor architecture with externalized plans enables localized correction and preserves prior progress during repair, suggesting a design pattern for improving GUI agent reliability.
The approach addresses a key limitation of current GUI agents—lack of user oversight—by making plans inspectable and revisable, which could influence future product design in automation tools.
Plover's design could reduce user frustration and increase trust in GUI automation, potentially leading to higher adoption in enterprise and consumer contexts.
Next signal: adoption of similar plan-centric architectures in commercial GUI automation products or further research on plan repair strategies.